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ABSTRACT 


The  Network  Centric  Warfare  approach  to  command  and  control  emphasizes 
decentralized  decision  making.  Consequently,  decision  makers  must  comprehend  and 
evaluate  infonnation  to  determine  the  optimal  course  of  action.  This  study  examines  how 
different  categories  of  uncertainty  (ambiguous/missing,  conflicting,  baseline)  and 
individual  differences  affect  response  time  in  decision  making  tasks.  The  researchers 
elicited  real-world  tactical  scenarios  from  veterans  of  Operation  Enduring  Freedom  and 
Operation  Iraqi  Freedom  in  which  uncertainty  was  present.  Nine  scenarios  were  given  to 
28  participants  at  the  Command  General  Staff  College,  FT  Leavenworth,  KS.  The 
participants  were  asked  to  make  a  decision;  their  responses  were  recorded  and  analyzed. 
The  results  indicate  that  the  category  of  uncertainty  and  scenario  difficulty  are  significant 
factors  in  determining  response  time.  No  individual  difference  factors  were  found  to  be 
significant.  These  findings  have  the  potential  to  improve  human  behavior  modeling, 
tactical  simulations,  and  representations  of  complex  task  environments. 
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EXECUTIVE  SUMMARY 


The  Information  Age  has  revolutionized  command  and  control  in  the  tactical 
environment.  The  advent  of  Network  Centric  Warfare  has  led  to  an  emphasis  on 
decentralized  decision  making  in  real-time.  The  warfighter  is  expected  to  evaluate  and 
analyze  large  quantities  of  information  to  produce  a  logical,  rational  decision  at  a 
moment’s  notice.  However,  warfare  often  is  neither  rational  nor  logical.  Although 
military  commanders  attempt  to  plan  fail-safe  operations,  uncertainty  often  will  create 
havoc  or  “fog”  in  the  battle  space.  This  fog  forces  commanders  to  adjust  their  plans  in 
real  time  by  using  the  information  available  to  them. 

To  apply  their  best  judgment,  commanders  must  utilize  various  inputs,  including 
machine  and  human  agents,  to  achieve  an  accurate  awareness  of  the  situation.  This 
reliance  on  both  human  and  machine  agents  strongly  suggests  a  Human  Systems 
Integration  (HSI)  approach  -  an  approach  that  emphasizes  the  symbiotic,  interdependent 
nature  of  human  and  machine  agents  to  solve  difficult  and  uncertain  problems. 

During  this  study,  U.S.  Army  officers,  with  recent  experience  in  Operation 
Enduring  Freedom  and  Operation  Iraqi  Freedom,  were  interviewed.  The  purpose  of  these 
interviews  was  to  elicit  real-world,  tactical  situations  where  uncertainty  affected  the 
decision  making  process.  These  scenarios  were  then  crafted  into  nine  vignettes 
representing  the  categories  of  uncertainty  (i.e.,  ambiguous/missing,  conflicting,  and 
baseline).  The  vignettes  were  presented  to  28  U.S.  Army  officers  at  FT  Feavenworth,  KS. 
The  participants  were  asked  to  determine  what  decision  they  would  make  for  each 
situation.  Their  decisions  and  response  times  were  recorded  and  analyzed.  Additionally, 
the  participants  completed  a  demographic  survey  and  the  Uncertainty  Response  Scale 
(e.g.,  emotional  uncertainty  &  cognitive  uncertainty).  The  researchers  hypothesized  that 
there  would  be  a  difference  between  the  categories  of  uncertainty  with  respect  to 
response  time.  Additionally,  it  was  believed  that  emotional  uncertainty,  cognitive 
uncertainty,  operational  experience,  and  response  time  would  be  significantly  correlated. 


xv 


The  results  suggest  that  the  category  of  uncertainty  coupled  with  scenario 
difficulty  were  significant  factors  in  determining  response  time.  Moreover, 
ambiguous/missing  scenario  response  times  were  significantly  greater  than  conflicting 
and  baseline  scenario  response  times.  Operational  experience,  cognitive  uncertainty, 
emotional  uncertainty,  and  response  time  were  not  significantly  correlated. 

In  conclusion,  the  findings  have  implications  for  advancing  the  state  of  the  art  in 
modeling  and  simulation.  This  research  provides  insights  into  human  behavior  under 
uncertain  conditions.  These  insights  can  aid  researchers  in  more  accurately  modeling 
human  behavior  in  tactical  environments.  The  findings  also  extend  to  civilian  and 
military  decision  making  under  complex  tasking. 


ACKNOWLEDGMENTS 


I  would  like  to  express  my  sincere  gratitude  to  several  individuals  involved  in  this 
18  month  project.  I  would  like  to  thank  LT  Derek  Mason,  LT  Douglas  Jones,  LT  Andrea 
Cassidy,  LT  Karen  Sankes,  and  LT  Patrick  Lazzaretti  for  their  dedication  and  diligence 
throughout  the  various  pilot  studies.  Next,  I  would  like  to  thank  MAJ  Michael  Martin  and 
Mr.  Duane  Riddle  of  the  U.S.  Army  Training  Doctrine  Command,  Monterey,  CA,  and  FT 
Leavenworth,  KS,  for  their  patience  and  support  in  coordinating  the  various  elements  of 
my  work.  I  have  deep  gratitude  for  the  guidance  and  mentoring  of  Dr.  Lawrence  G. 
Shattuck,  thesis  advisor,  and  Dr.  M.  Quinn  Kennedy,  thesis  co-advisor.  They  were 
instrumental  in  articulating  the  essence  of  the  study  and  provided  motivation  for  a  strong 
finish  in  the  last  phases  of  my  thesis.  Additionally,  I  would  also  like  to  thank  Dr.  Nita  L. 
Miller  for  her  insight  and  experience  in  the  creation  and  execution  of  my  research. 
Finally,  and  most  importantly,  I  would  like  to  thank  my  wife,  Elizabeth  Kemmerer,  for 
being  empathetic  and  supportive  while  bringing  a  sense  of  humor  to  my  quirkiness  and 
obsessive  behavior  concerning  my  work.  I  love  you  and  thank  you. 


xvii 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


I.  INTRODUCTION 


History  is  captured  by  moments  of  uncertainty.  From  the  ancient  and  mythical 
battlefields  of  the  Peloponnese  to  the  rugged  and  unpredictable  terrain  of  the  War  on 
Terror,  statesmen,  politicians,  and  military  commanders  have  made  critical  decisions  with 
uncertain  information.  This  is  the  nature  of  the  world  we  live  in;  very  few  things  are 
certain,  and  most  things  are  uncertain. 

This  natural  phenomenon  has  prompted  many  philosophers  and  theorists  to 
address  and  pontificate  on  uncertainty’s  role  in  the  natural  world.  Consequently, 
psychologists,  historians,  and  researchers  have  approached  uncertainty  from  cognitive, 
historical,  and  sociologists’  perspectives.  More  importantly,  they  have  attempted  to 
define  uncertainty,  detennine  the  effects,  and  discover  its  origins.  Their  research  has 
yielded  varying  and  numerous  results.  Regardless,  their  research  has  established  a  great 
foundation  for  future  inquiry,  and  for  this  study,  the  effects  of  uncertainty  on  decision¬ 
making. 

A.  THE  HISTORICAL  PREMISE  OF  UNCERTAINTY 

Commentary  on  uncertainty  can  be  traced  back  to  the  greatest  thinkers  in  Western 
literature  and  philosophy.  Although  the  constructions  of  their  arguments  do  not  attempt 
to  study  uncertainty,  they  do  make  reference,  and  find  its  presence  important  in  their 
works.  In  Discourse  of  Method,  Rene  Descartes  (1637)  addressed  the  importance  of 
discovery  and  truth  through  science.  To  accomplish  this,  he  asked  man  to  “reject  all 
previous  knowledge,  opinion,  and  customs”  (Descartes,  1637,  p.  21).  He  writes: 

The  first  was  never  to  accept  anything  as  true  which  I  did  not  clearly  know 
to  be  such,  that  is  to  say,  carefully  to  avoid  haste  and  prejudgment,  and  to 
accept  nothing  as  true  except  what  was  presented  to  my  mind  so  clearly 
and  distinctly  as  to  exclude  all  possibility  of  doubt. 

Descartes  was  arguing  two  points.  First,  knowledge,  and  therefore  truth,  cannot, 
and  must  not,  be  concluded  without  a  thorough  investigation  of  the  question  posed. 
Second,  a  statement  cannot  be  considered  true  unless  all  uncertainty  (i.e.,  doubt)  is 
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removed.  The  latter  of  the  two  is  most  important  for  this  debate,  because  Descartes  is 
advocating  that  doubt  (i.e.,  uncertainty)  naturally  exists  in  any,  and  all,  scientific  and 
intellectual  arguments. 

Moreover,  Thomas  Hobbes  (1651)  discussed  this  innate  uncertainty  through  the 
nature  of  man.  In  The  Leviathan,  the  Englishman  examined  the  relationship  between 
knowledge,  opinion,  judgment,  and  faith.  Hobbes  writes,  “no  discourse  whatsoever  can 
end  in  absolute  knowledge  of  fact,  past,  or  to  come. .  .no  man  can  know  by  discourse,  that 
this,  or  that,  is,  has  been,  or  will  be;  which  is  to  know  absolutely”  (p.  511).  Therefore,  the 
search  for  understanding  can  never  conclude  in  concrete  truth,  because  man  cannot 
adequately  predict  the  future  or  what  is  to  come.  More  importantly,  his  judgment  and 
reason  can  be  clouded  by  personal  conscious  and  faith. 

However,  there  is  something  more  important  to  be  learned  from  the  writings  of 
Descartes  and  Hobbes.  The  philosophers  articulate  the  symbiosis  of  the  natural  state  of 
man  and  the  events  of  history.  Although  this  may  seem  trivial,  it  supports  the  premise  of 
uncertainty  in  the  modem  world;  because  man  is  naturally  flawed,  favored  by  opinion 
and  prejudgment,  unable  to  grasp  all  possible  knowledge,  and  his  affairs  dynamic, 
uncertainty  will  always  be  present  in  all  of  man’s  institutions.  Thus,  economics,  politics, 
and  for  this  study,  warfare,  by  nature,  will  be  laced  with  uncertainty.  Consequently,  it 
must  be  studied  and  understood. 

B.  UNCERTAINTY  IN  WARFARE 

The  role  of  uncertainty  in  warfare  has  been  documented  throughout  history. 
Recently,  Secretary  of  Defense  Donald  Rumsfeld,  trying  to  explain  the  clouded  future  of 
the  War  on  Terror,  delivered  a  set  of  shrouded  words  that  left  the  media  confused  and 
hysterical.  In  under  a  minute,  the  battle -hardened  patriot  described  the  innate 
characteristics  of  war. 

There  are  things  that  we  know.  There  are  known  unknowns.  That  is  to  say, 
there  are  things  that  we  know  we  don’t  know.  But  there  are  also  unknown 
unknowns.  There  are  things  we  don’t  know  we  don’t  know  (Rumsfeld, 

2002). 
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Although  many  scoffed,  laughed,  and  viewed  his  uncanny  remarks  as  mysterious, 
the  proposed  message  was  intended  to  prepare  and  warn  the  American  public  of  the 
complex,  dynamic  nature  of  warfare.  Like  Hobbes’  interpretation  of  human  nature, 
Rumsfeld  viewed  uncertainty  as  an  inherent  attribute  of  warfare. 

However,  Rumsfeld’s  often  criticized,  yet  sometimes  revered,  speech  was  not  the 
first  to  describe  the  assumptions,  estimates,  and  uncertain  nature  of  armed-conflict.  More 
than  two  millennia  earlier,  military  philosopher  Sun  Tzu  (650)  argued  that  the  factors  of 
war  can  be  controlled;  therefore  uncertainty  and  chaos  can  be  spawned,  developed,  and 
adjusted  to  confuse  an  opponent.  Victory,  therefore,  can  be  achieved  by  creating  a 
specific  amount  of  uncertainty  that  both  paralyzes  and  strikes  fear  in  one’s  adversary. 

Conversely,  in  the  19th  century,  military  theorist  Carl  von  Clauswitz  (1805) 
argued  that  uncertainty  is  not  a  variable  that  can  be  manipulated  in  armed-conflict. 
Instead,  he  argued  uncertainty  to  be  a  powerful  and  uncontrollable  force  that  man  is 
incapable  of  harnessing.  Clauswitz  described  the  climate  of  war  using  four  distinct 
characteristics:  danger,  exertion,  uncertainty,  and  chance.  He  wrote,  “war  is  the  realm  of 
uncertainty;  three  quarters  of  the  factors  on  which  action  in  war  is  based  are  wrapped  in  a 
fog  of  greater  or  lesser  uncertainty”  (Clauswitz,  1805,  p.  102).  Consequently,  uncertainty 
creates  “friction”  that  makes  the  simplest  movements  difficult.  To  Clauswitz,  war  is  not 
solely  an  act  of  pure  intellect  or  intuition,  but  rather  a  delicate  congruence  of  fate,  luck, 
and  skill. 

Interestingly,  Sir  Jillian  Corbett  (1911)  may  have  best  expressed  the  unique  nature 
of  man  and  uncertainty  in  war.  A  student  of  Clauswitz  and  Jomini,  the  British  historian 
disagrees  with  the  idea  that  war  is  an  art  vice  a  science  with  “no  fixed  laws  or  rules” 
(Corbett,  191 1,  p.  8).  Instead,  Corbett  argues  that  history  shows  “certain  lines  of  conduct 
produce  certain  effects”  (p.  8).  However,  Corbett  later  suggests  that  “strategic  analysis 
can  never  give  exact  results... only  at  approximations...”  and  will  always  “leave  much  to 
judge”  (p.  83). 
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His  statements  give  more  credence  to  Clauswitz’s  views;  that  war  is  not  an 
equation  of  set  variables,  but  rather  an  open  system  of  ever  changing  inputs  and  outputs. 
Therefore,  uncertainty  will  always  find  a  place  among  the  commander’s  sure  plans  or  the 
infantryman’s  charge. 

C.  HUMAN  SYSTEMS  INTEGRATION 

To  explore  the  role  of  uncertainty  in  military  decision  making,  our  study  will 
address  the  topic  from  the  perspective  of  Human  Systems  Integration  (HSI).  The 
definition  of  HSI  continues  to  be  the  subject  of  debate.  However,  two  definitions  best 
describe  the  essence  of  the  discipline.  First,  HSI  is  “the  interdisciplinary  approach  that 
makes  elicit  the  underlying  tradeoffs  across  the  domains,  facilitating  optimization  of  total 
system  performance,”  (Miller  &  Shattuck,  2006).  Second,  HSI  is  “a  human  centered 
approach  to  the  design,  development,  and  acquisition  of  systems  from  cradle  to  grave,” 
(Read,  2007,  p.  3).  In  simpler  terms,  HSI  aims  to  incorporate  the  human  system  in  the 
“systems  of  systems”  architecture;  it  attempts  to  remedy  an  individual’s  weaknesses  and 
highlight  their  strengths.  In  order  to  do  so,  HSI  focuses  on  the  eight  domains  and  the 
tradeoffs  within  each.  The  following  are  the  eight  domains  of  HSI: 

•  Manpower 

•  Training 

•  Personnel 

•  Survivability 

•  Habitability 

•  Human  Factors  Engineering 

•  Health  Hazards 

•  System  Safety 

By  incorporating  these  eight  domains  early  in  the  acquisition  cycle,  the  U.S. 
Department  of  Defense  hopes  to  reduce  costs,  human  survivability,  and  system  lifespan. 
Further  explanation  of  these  tenns  can  be  found  in  Department  of  Defense  Instruction 
5000.2  (U.S.  Department  of  Defense,  2003)  or  Handbook  of  Human  Systems  Integration 
(Booher,  2003). 


4 


For  the  scope  of  this  study,  the  exploration  of  uncertainty  in  tactical  decision 
making  will  be  focused  on  modeling  and  simulation.  Therefore,  close  attention  will  be 
paid  to  the  domains  of  human  factors  engineering,  personnel,  and  training.  The 
researchers’  hope  is  to  create  algorithms  for  decision  making  under  uncertainty  in  tactical 
simulations,  thus  providing  a  more  realistic  interpretation  of  the  actual  events  of  the 
battle  space. 

D.  RESEARCH  QUESTIONS 

The  historical  presence  of  uncertainty  and  its  influence  on  military  history  has 
prompted  several  important  questions.  While  these  questions  provide  an  adequate 
foundation  for  research  in  the  field  of  uncertainty,  we  must  remind  ourselves  that  the 
problem  reaches  far  into  the  battle  space.  When  addressing  the  problem  from  the 
perspective  of  HSI,  the  machine  and  human  must  be  discussed  in  integrated,  symbiotic 
tenns.  The  following  research  questions  were  not  mutually  exclusive  of  the  human  being, 
but  instead  created  to  better  understand  the  human’s  cognitive  abilities  in  combat. 

•  What  is  uncertainty?  What  is  its  origin?  What  is  its  definition? 

•  How  does  uncertainty  affect  the  man  and  machine  symbiosis? 

•  What  models  and  taxonomies  can  aid  in  the  understanding  of  uncertainty 
in  HSI? 

•  What  are  the  relationships  between  uncertainty,  data,  and  individual 
difference  factors? 

•  Does  an  individual’s  emotional  or  cognitive  response  to  uncertainty  affect 
his/her  ability  to  cope  with  uncertainty? 

•  Do  decision  makers  delay  their  decisions  when  faced  with  uncertainty? 

E.  RESEARCH  OBJECTIVES 

With  the  premise  properly  stated  and  research  questions  posed,  the  objectives  for 
the  study  can  be  outlined: 

•  Create  a  methodology  for  examining  uncertainty  in  the  tactical 
environment. 

•  Emphasize  the  importance  of  HSI  in  exploring  uncertainty  across  the 
battle  space. 

•  Begin  to  identify  the  individual  difference  factors  that  influence  decisions 
under  uncertain  conditions. 
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•  Develop  of  algorithms  for  modeling  and  simulation. 

•  Provide  a  clear  direction  for  future  research  in  tactical  decision  making 
under  uncertainty. 

F.  THESIS  ORGANIZATION 

The  organization  of  this  thesis  follows  a  traditional  format.  In  accordance  with 
APA  standards,  it  documents  a  literature  review,  methodology,  results  chapter,  and 
discussion.  Chapter  II  uses  a  top-down  approach  to  examine  decision-making  under 
uncertainty  through  the  perspective  of  HSI.  Literature  regarding  command  and  control 
(C2),  decision-making  models,  uncertainty,  personality  characteristics,  and  experience 
are  analyzed  and  discussed  to  better  explain  their  role  in  the  realm  of  the  uncertainty. 

Chapter  III  focuses  on  the  previous  pilot  studies  perfonned  at  NPS,  along  with  the 
methodology  used  to  conduct  the  final  study.  The  pilot  studies  provide  an  interesting  and 
important  tale  when  addressing  the  nature  of  uncertainty.  Additionally,  “lessons  learned” 
are  documented  and  translated  into  future  success.  Moreover,  a  detailed  description  of  the 
methodology  is  provided  for  future  replication. 

Chapter  IV  and  Chapter  V  provide  statistical  analysis  and  a  discussion  of  the 
results.  Additionally,  conclusions  based  on  previous  research  will  be  documented,  along 
with  opportunities  for  future  research  in  the  field  of  uncertainty. 
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II.  LITERATURE  REVIEW 


A.  DECISION  MAKING  AND  COMMAND  AND  CONTROL 

1.  Network-Centric  Warfare 

To  remedy  the  effects  of  uncertainty,  militaries  have  established  means  of 
communications  and  procedures  to  relay  plans,  monitor  execution,  and  provide 
assessment  (or  assess  results).  These  means  of  command  and  control  (C2)  remain  central 
and  increasingly  important  to  all  military  operations  and  allow  commanders  to  issue 
explicit  and  implicit  orders  when  executing  simple  and  complex  operations. 

The  theory  of  C2  has  evolved  over  the  course  of  military  history.  However,  the 
Industrial  Age  forever  altered  its  essence.  The  20th  century  ushered  in  the  use  of  machines 
in  the  battle  space.  Ships,  aircraft,  and  tanks  greatly  increased  the  range,  distance, 
lethality,  and  speed  of  battle.  Additionally,  radio  and  radar  allowed  commanders  to  view 
the  enemy  at  great  distances  and  communicate  over  thousands  of  miles  of  terrain  and 
ocean.  Consequently,  platfonn-centric  warfare  became  the  preferred  method  of  C2 
operations  (i.e.,  the  commander  makes  a  centralized  decision  and  communicates  his 
direction  to  subordinate  platforms  who  then  execute  the  commanders’  orders)  (Alberts, 
Garstka,  &  Stein,  1999;  Alberts,  2003;  Cebrowski  &  Garstka,  1998).  More  importantly, 
the  warfighters  were  no  longer  autonomous  in  their  own  thought  or  action,  but  rather 
guided,  and  sometimes  aided  or  instructed  by  mechanized  technology  (i.e.,  the  integration 
of  the  human  and  machine  system  were  vital  to  operational  success). 

The  evolution  of  technology  (e.g.,  satellite  communication,  AEGIS,  and 
simulation)  soon  gave  new  capabilities  to  commanders  and  operators.  For  the  first  time  in 
warfare,  the  human  possessed  the  capability  to  share,  analyze,  disseminate,  and 
internalize  distant  battle  space  infonnation  in  near  real-time.  Consequently,  a  new  vision 
of  military  C2  operations  emerged,  and  Cebrowski  and  Garstka’s  (1998)  Network  Centric 
Warfare  (NCW)  theory  changed  the  direction  of  C2  and  military  decision-making. 
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The  NCW  concept  has  revolutionized  military  operations.  The  centralized 
information  and  decision-making  environment  of  the  past  is  being  replaced  by  a  shared 
network,  focused  on  decentralized  decision-making.  Consequently,  DeLange  (2006) 
argues  that  NCW  is  merely  a  named  process  and  the  underlying,  important  message  is 
“Decision  Centric  Warfare”  (DCW)  (i.e.,  improved  decision-making  is  the  most 
important  aspect  behind  the  NCW  revolution).  But  while  the  particulars  and  details  of 
NCW  operations  are  fascinating  and  intriguing,  the  most  important  concepts  lie  in  its  four 
domains  (cognitive,  social,  physical,  and  information)  outlined  by  its  creators  (Cebrowski 
&  Garstka,  1998).  More  importantly,  for  this  study,  how  the  cognitive  domain  impacts 
Network  Centric  Operations  (NCO)  when  clouded  by  uncertainty 

2.  The  Cognitive  Domain 

The  cognitive  domain  is  the  mind  of  the  decision  maker  (Alberts,  Garstka,  Hayes, 
&  Signori,  2001).  It  serves  as  the  most  important  element  in  NCO.  While  automation  in 
complex  systems  has  increased  over  the  last  quarter  century,  the  decision  maker 
continues  to  play  the  most  important  role  in  the  system.  Consequently,  the  ultimate 
success  of  NCW  relies  on  the  integration  of  human  cognition  and  technological  capability 
(Read,  2007,  &  Baker,  2002). 

To  better  understand  the  capabilities  and  limitations  of  human  cognition  in  NCO, 
Garstka  and  Alberts  (2004)  have  described  the  human’s  thought  process  as 
“sensemaking,”  while  also  placing  an  emphasis  on  “shared  sensemaking”  or 
“collaboration.”  The  latter  articulates  two  fundamental  ideas:  (1)  the  cognitive  domain  is 
influenced  by  the  social  domain  (i.e.,  culture,  organization,  politics),  and  (2)  the  decision 
makers  share  their  evaluation  of  the  situation  with  their  peers  (Garstka  &  Alberts,  2004). 
Consequently,  an  individual  decision  maker  can  have  a  major  impact,  through  a  pseudo¬ 
social  medium,  on  the  course  of  action  at  both  the  micro  and  macro  level  due  to  the 
ability  to  share  and  analyze  information  in  real  time.  Additionally,  Caneva  (1999) 
suggests  NCW  has  the  potential  to  not  only  increase  military  operations  spatially,  but  also 
temporally.  Therefore,  the  cognitive  domain  can  expand  the  battlefield  beyond  our  own 
expectations  and  imagination. 
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While  these  points  are  important  to  recognize,  it  is  outside  the  scope  of  our 
research.  Instead,  our  goal  is  to  focus  on  the  “sensemaking”  occurring  at  the  individual 
cognitive  level  prior  to  peer-sharing.  To  describe  the  aforementioned,  Garstka  and 
Alberts  (2004)  explain  the  development  of  mental  models  to  facilitate  decision-making. 
Additionally,  the  researchers  argue,  “over  time  and  with  experience,  people  build  up  a 
repertoire  of  mental  models  that  apply  across  a  range  of  situations”  (Garstka  &  Alberts, 
2004,  p.  30).  Their  researcher  implicitly  hints  at  the  various  thought  processes  applied  to 
decision-making. 

B.  DECISION-MAKING  MODELS 

To  describe  how  individuals  make  or  should  make  decisions,  researchers  have 
developed  three  models:  (1)  normative,  (2)  descriptive,  and  (3)  naturalistic.  Although  all 
three  models  describe  the  human  thought  process  in  unique  and  explicit  terms,  the 
naturalistic  model,  or  naturalistic  decision-making  model  (NDM),  has  been  widely 
accepted  in  recent  years  due  to  its  utility  in  operational  settings  (Shattuck,  2007).  More 
importantly,  the  NDM  characteristics  outlined  by  Orasanu  and  Connolly  (1993)  (e.g.,  ill- 
structured  problems,  uncertain  dynamic  environments,  multiple  players,  and 
organizational  norms  and  goals)  describe  the  situations  and  context  of  military  exercises 
and  engagements  (e.g.,  the  recent  2008  military  showdown  between  the  United  States  and 
Iranian  naval  forces  in  the  Straits  of  Hormuz). 

1.  Naturalistic  Decision  Making 

In  examining  the  utility  of  NDM  in  operational  settings,  Zsambok  and  Klein’s 
(1997),  Naturalistic  Decision  Making,  cites  several  real-world,  research  studies.  For 
example,  Hutchins  (1997)  addresses  human  error  and  situation  awareness  in  U.S.  Navy 
Littoral  Anti  Air  Warfare  (AAW)  operations,  while  Serfaty,  MacMillan,  Entin,  and  Entin 
(1997)  examined  the  importance  of  expertise  in  military  tactical  decision-making. 

Coincidentally,  the  AEGIS  training  center’s  Tactical  Decision  Making  under 
Stress  (TADMUS)  program  focused  on  decision-making  in  stressful  and  time-sensitive 
environments  (Cannon-Bowers  &  Salas,  1998).  Additionally,  Kaempf  and  Klein  (1993) 
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examined  the  decision-making  strategies  of  naval  officers  aboard  U.S.  Navy  AEGIS 
Cruisers.  The  aggregate  research  illustrates  the  complex,  uncertain  environments  in 
which  decision  makers  must  operate. 

Consequently,  NDM  researchers  have  developed  overlapping  theories  on  how 
decision  makers  formulate  and  select  their  courses  of  action.  For  example,  Klein’s  (1993) 
Recognition  Primed  Decision-Making  (RPD)  model  argues  that  decision  makers  rely  on 
past  experiences  to  select  their  courses  of  action  (i.e.,  individuals  do  not  select  from 
several  different  outcomes,  but  rather  choose  their  course  of  action  based  on  past 
experience)  (see  Figure  1).  He  describes  the  model  as  consisting  of  two  fused  parts:  (1) 
situational  assessment  and  (2)  mental  simulation  (Klein,  1993).  Klein  argues  mental 
situational  assessment  generates  a  plausible  course  of  action  (COA)  and  mental 
simulation  evaluates  that  COA.  However,  it  is  important  to  note,  that  Klein  (1989,  as 
cited  in  Wickens,  Lee,  Liu,  and  Becker,  2004)  describes  the  RPD  model  as  a  “more 
refined  description”  of  the  various  components  of  Rassmussen’s  (1983,  1986,  1993) 
Skill,  Rule,  and  Knowledge  (SRK)  model  (p.  173).  However,  while  the  aforementioned 
models  describe  the  process  behind  decision  making,  they  do  not  adequately  add  the 
important  aspect  of  perception. 


Figure  1.  Klein’s  Recognition  Primed  Decision  Model  (1993). 
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2.  Perception 

Endsley  (1995)  discusses  perception  in  terms  of  situation  awareness  (SA).  She 
defines  SA  as  the  “perception  of  the  elements  of  in  the  environment  within  a  volume  of 
time  and  space,  the  comprehension  of  their  meaning  and  the  projection  their  status  to  the 
near  future”  (Endsley,  1995,  p.  36).  Consequently,  the  information  presented  to  the 
commanders  can  be  radically  altered  by  their  situational  perception. 

To  remedy  this  gap,  Wickens  et  al.  (2004)  offers  an  integrated,  adaptive  decision 
making  model  (see  Figure  2).  Although  the  model  effectively  integrates  RPD,  SA,  and 
SKR,  it  fails  to  address  the  symbiosis  of  man  and  machine  in  the  battle  space  (i.e.,  the 
advent  of  technology  discussed  in  Chapter  II. A.  1  has  made  the  two  parties  mutually 
dependent  of  each  other).  Consequently,  when  discussing  the  role  of  the  warfighter  in 
“sensemaking,”  one  must  apply  a  model  that  incorporates  both  man  and  machine.  The 
model  that  best  portrays  this  necessary  relationship  is  the  Dynamic  Model  of  Situation 
Cognition  (DMSC)  (Shattuck  &  Miller,  2006;  Miller  &  Shattuck,  2006). 
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Figure  2.  Wickens  et  al.  Integrated  Model  (2004)  . 
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C.  DYNAMIC  MODEL  OF  SITUATION  COGNITION 

The  DMCS  represents  the  integration  of  man  and  machine  in  the  battle  space  (see 
Figure  3).  It  shows  the  processes  associated  with  human-machine  interaction  when 
operating  in  complex,  fast-paced,  and  uncertain  environments.  Additionally,  it  describes 
the  perception,  or  lens,  of  the  warfighter  when  making  critical,  tactical  decisions.  As 
described  in  Chapter  I.A,  because  individuals  are  flawed  and  unable  to  grasp  all 
knowledge  they  may  naturally  interpret  information  differently.  Lastly,  it  addresses  the 
inherent  uncertainty  involved  in  cognitive  reasoning  when  conducting  real-world 
operations. 

1.  Principles  of  DMSC 

DMSC  contains  two  distinct  and  important  principles:  (1)  ground  truth  will  never 
be  completely  realized  or  recognized  by  the  decision  maker  and  (2)  the  machine  or 
system  cannot  identify  or  articulate  uncertainty  through  its  medium.  Both  principles  are 
the  curse  of  the  infonnation  technology  age.  Because  society  is  obsessed  with  the  idea  of 
perfection,  we  strive  to  create  no  doubt  or  uncertainty  in  our  world.  We  have  been 
conditioned  to  assume  the  technology  cannot  be  wrong;  it  is  the  antidote  to  fix  our 
inherent  flaws.  Consequently,  we  allow  ourselves  to  be  drawn  into  the  perfect  storm,  or 
as  Miller  and  Shattuck  (2006)  profoundly  call  it,  “the  seduction  of  technology.” 

This  is  no  more  evident  than  in  the  military,  where  we  often  struggle  to  remedy 
the  effects  of  uncertainty  in  the  tactical,  operational  environment.  Even  worse,  the 
physical  representation  of  uncertainty  continues  to  present  an  enormous  challenge  in  the 
modeling  and  simulation  discipline.  Regardless,  before  we  can  study  the  effects  of 
uncertainty  on  decision  making,  we  must  first  identify  how  the  uncertainty  manifests  in 
both  the  technology  and  human  systems.  Fortunately,  DMSC  answers  this  critical 
question. 
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Figure  3.  Dynamic  Model  of  Situated  Cognition  (Shattuck  &  Miller,  2006). 

2.  Uncertainty  Emergence 

The  DMSC  shows  the  presence  of  uncertainty  throughout  the  model.  It  appears  as 
early  as  Oval  2,  and  influences  the  decision  making  process.  As  Shattuck  and  Miller 
(2006)  note,  “the  specific  comprehension  achieved  by  the  decision  maker  is  a  function  of 
the  data  that  have  propagated  through  the  model  and  the  contents  of  the  lens,”  (p.  998). 
Consequently,  Oval  (5)  and  Oval  (6)  show  uncertainty’s  effect  on  the  decision  maker’s 
comprehension  and  projection.  However,  before  we  can  discuss  uncertainty  in  the  context 
of  the  model,  we  must  explore  the  ovals  where  it  manifests. 

a.  Oval  1 

Oval  1  represents  ground  truth;  ground  truth  being  the  absolute, 
unequivocal  truth  as  described  by  Descartes  (1637). 

b.  Oval  2 

Oval  2  is  how  accurately  technology  depicts  ground  truth.  Technological 
accuracy  is  determined  by  the  efficiency  of  applied  sensors  (e.g.,  if  the  sensors  do  not 
detect  various  units  in  the  battle  space,  then  the  unequivocal  representation  of  ground 
truth  is  lost)  (Shattuck  and  Miller,  2006).  Additionally,  the  data  can  be  misidentified. 
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Shattuck  and  Miller  suggest  that  in  a  perfect  world  Ovals  1  and  2  would  be  mirror 
images,  thus  allowing  for  the  decision  maker  to  view  the  world  from  “God’s  eye.” 

c.  Oval  3 

Oval  3  describes  the  transition  from  the  sensors  to  the  display.  Shattuck 
and  Miller  (2006)  note  that  current  C2  systems  (Blue  Force  Tracker,  Tactical  Tomahawk 
Weapons  Control  System)  allow  for  decision  makers  to  tailor  their  displays  to  view 
perceived  important  information.  Consequently,  some  information  may  not  be  seen  or 
processed.  Additionally,  faulty  algorithms  may  misinterpret  what  is  ground  truth,  thus 
depicting  an  inaccurate  picture. 

d.  Oval  4 

The  transition  from  Oval  3  to  Oval  4  represents  the  bridge  between 
technology  and  the  operator  (hence,  human  systems  integration).  Oval  4  shows  the  data 
perceived  by  the  decision  maker.  Shattuck  and  Miller  (2006)  describe  this  process  in 
tenns  of  passive  or  active  input.  Active  input  is  a  result  of  the  decision  maker  requesting 
or  “pulling”  infonnation,  where  passive  involves  the  information  that  is  not  requested  or 
“pushed”  upon  the  decision  maker  (Shattuck  &  Miller,  2006). 

e.  Oval  5 

Oval  5  represents  the  comprehension  of  the  decision  maker.  Endsley 
(1995)  (as  cited  in  Shattuck  and  Miller,  2006)  describes  comprehension  as  multiple 
processes,  including  integration,  analysis,  and  interpretation.  Additionally,  Oval  5  is  the 
result  of  Lens  B  in  the  model.  Lens  B  articulates  several  factors  including  personality 
states  and  traits,  experience,  and  social  culture  that  influence  the  comprehension  of  the 
decision  maker  (Shattuck  &  Miller,  2006).  Elements  of  Lens  B  will  be  discussed  in 
Chapter  II.F. 2. 
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f.  Oval  6 

Oval  6  embodies  the  idea  of  projection.  Shattuck  and  Miller  (2006)  define 
projection  as  “the  prediction  of  the  decision  maker,”  (p.  998).  At  this  point,  decision 
makers  have  ascertained  what  they  believe  to  be  true,  and  in  some  cases,  have  transmitted 
their  depiction  or  beliefs  about  the  situation  to  other  decision  makers  in  the  battle  space. 

g.  Amorphous  Shapes 

The  amorphous  shapes  surrounding  Oval  5  and  Oval  6  represent  the 
various  organizations  or  interpretations  of  the  information  (Shattuck  &  Miller,  2006). 
Although  the  same  infonnation  may  be  presented  to  two  different  decision  makers,  their 
interpretation  of  the  information  may  be  slightly  or  very  different.  Consequently,  the 
decision  makers  may  fonnulate  different  mental  models  of  the  situation  (Shattuck  & 
Miller,  2006).  Additionally,  the  enlarging  of  the  amorphous  shapes  from  Oval  5  to  Oval  6 
depicts  the  idea  that  there  is  more  uncertainty  involved  in  projection  vice  comprehension. 
A  prediction  of  the  future  is  merely  one  possible  outcome  of  numerous  possibilities  for 
the  future;  the  farther  an  individual  predicts  into  the  future,  the  more  uncertainty  will  be 
present  (Shattuck  &  Miller,  2006). 

3.  Feedback  Loops 

To  curtail  uncertainty  in  the  DMSC,  Shattuck  and  Miller  (2006)  emphasize  the 
iterative,  dynamic  process  inherent  in  the  model.  Feedback  loops  depict  decisions  or 
reorientations  by  the  decision  maker  (see  Figure  4).  Thus,  the  individual  can  mitigate 
uncertainty  by  constantly  updating  their  cognitive  approach  or  the  technologies’  output. 
Unfortunately,  in  most  cases,  the  operator  cannot  simply  “tag”  an  object  as  uncertain  or 
ambiguous.  However,  they  can  inform  other  operators  of  the  incorrect  information  being 
displayed.  This  leads  to  the  next  important  point  in  terms  of  uncertainty. 
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Figure  4.  Dynamic  Model  of  Situated  Cognition  Feedback  System  (Shattuck  &  Miller, 

2006). 

Because  NCW  relies  on  a  shared  picture,  or  common  operating  picture  (COP),  the 
propagation  of  uncertainty  in  the  battle  space  can  be  disastrous  regardless  of  the  echelon 
of  command  (i.e.,  uncertainty  experienced  by  a  flag  officer  or  junior  enlisted  soldier  can 
have  equal  effect).  This  is  why  the  recent  wars  in  Afghanistan  and  Iraq  have  focused  on 
young  soldiers  making  real  time  decisions  with  significant  consequences.  However,  the 
impact  of  uncertainty  up  and  down  the  chain  of  command  is  a  pervasive  issue  and  will 
need  to  be  explored  in  future  research. 

The  present  research  examined  tactical  decisions  made  by  middle  echelons 
commanders  (e.g.,  company  commanders  (CO  CDR)).  By  doing  so,  the  researchers 
established  a  basic  understanding  of  the  role  of  uncertainty  in  their  decisions.  More 
importantly,  they  examined  how  uncertainty  influences  the  time  it  takes  to  make  a 
decision  and  how  individual  differences  help  or  hinder  their  responses. 

But  before  we  can  begin  to  study  uncertainty,  we  must  (1)  define  uncertainty,  and 
(2)  describe  the  individual  differences  that  influence  our  decisions  under  uncertain 
conditions.  While  these  tasks  seem  simple  the  answer  is  as  ambiguous  as  the  word  being 
examined. 
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D.  DEFINITION  OF  UNCERTAINTY 


Researchers  continue  to  argue  the  definition  of  uncertainty.  In  capturing  the  mood 
of  the  debate,  Downey  and  Slocum  (1975,  p.  562,  quoted  in  Milliken,  1987,  p.  134)  state 
the  overabundant  use  of  the  term  “uncertainty”  has  made  it  “all  too  easy  to  assume  that 
one  knows  what  he  or  she  is  talking  about.”  Regardless,  Lipshitz  and  Strauss  (1997) 
propose  uncertainty  “in  the  context  of  action  is  a  sense  of  doubt  that  blocks  or  delays 
action”  (p.  150).  Additionally,  Lipshitz  and  Strauss  distinguish  uncertainty  from 
ambiguity.  Paraphrasing  Hogarth  (1987),  the  researchers  describe  ambiguity  as  “lacking 
precise  knowledge  about  the  likelihood  of  events”  (Lipshitz  and  Strauss,  p.  150). 

Conversely,  Cohen  and  Freeman  (1996)  use  a  quantitative  approach  to 
characterize  uncertainty.  The  researchers  define  uncertainty  as  “probabilities  other  than 
zero”  (Cohen  &  Freeman,  p.  179).  Additionally,  Conrath  (1967)  constructed  a  one¬ 
dimensional  scale  using  the  terms  “certainty”  and  “uncertain”  as  the  two  extreme 
opposite  points  along  the  continuum.  However,  McCloskey  and  Klein  (1996)  go  beyond 
defining  the  term,  and  identify  four  categories  of  uncertainty:  (1)  missing  information,  (2) 
unreliable  information,  (3)  complex  infonnation,  and  (4)  ambiguous/conflicting 
information. 

McCloskey  and  Klein  (1996)  define  information  as  ambiguous  or  conflicting  “if 
there  is  more  than  one  reasonable  way  to  interpret  it”  (p.  195).  However,  the  researchers 
advocated  a  different  approach.  Rather  than  keeping  ambiguous/conflicting  information 
as  a  pair,  they  separated  them  into  two  different  entities;  ambiguous  and  conflicting. 
Then,  from  the  pilot  studies  outlined  in  Chapter  III. A,  they  adjoined  missing  and 
ambiguous.  Additionally,  the  researchers  viewed  complex  information  as  product  of  the 
other  defined  categories  of  uncertainty  (i.e.,  the  more  uncertain  a  situation  becomes,  the 
more  complexity  involved).  The  modification  of  the  categories  of  uncertainty  reduced  the 
number  of  categories  to  three.  Finally,  the  researchers  added  the  category  of  baseline  to 
establish  situations  where  little  or  negligible  uncertainty  appeared.  Thus,  the  researchers 
viewed  the  categories  of  uncertainty  as  the  following:  (1)  ambiguous/missing 
information,  (2)  conflicting  infonnation,  (3)  unreliable  infonnation,  and  (4)  baseline 
information. 
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E.  CATEGORIES  OF  UNCERTAINTY 

The  categories  of  uncertainty  provide  structure  for  examining  its  affects  in  the 
natural  world.  However,  before  we  can  adequately  study  the  levels  of  uncertainty,  we 
must  explicitly  define  the  four  terms.  However,  this  research  will  only  address  three  of 
the  categories  of  uncertainty  (ambiguous/missing,  conflicting,  and  baseline).  Examples 
are  used  to  illustrate  the  differences  between  the  aforementioned  tenns. 

1.  Ambiguous/Missing  Information 

Ambiguous/Missing  data  represents  information  that  may  be  unavailable,  but  if 
available  may  be  unclear  or  not  critical  to  the  decision.  In  other  words,  the  decision 
maker  may  have  information  that  is  unnecessary  to  make  an  adequate  decision,  or  may 
possess  information  that  is  very  muddled  and  not  explicit.  Furthermore,  the  data  may 
result  in  a  choice  between  numerous  alternatives.  This  last  statement  coincides  with 
McCloskey  and  Klein’s  (1996)  interpretation  of  complex  information. 

2.  Conflicting  Information 

Conflicting  data  represents  multiple  sources  of  information  that  is  contradictory  in 
nature.  Mathematically  this  is  expressed  by  x  is  true,  y  is  true,  but  x  equals  y  is  not  true. 
An  example  is  provided  later  in  this  chapter. 

3.  Unreliable  Information 

Unreliable  data  consists  of  information  where  the  source  possess  very  little  or  no 
credibility  (McCloskey  &  Klein,  1996).  The  decision  maker  does  not  trust  the  source,  or 
feels  the  source  contains  low  fidelity.  This  type  of  information  is  the  link  between  the 
technological  and  warfighter  sides  of  the  DMSC,  because  it  is  dependent  on  whether  the 
operator  views  the  system  or  the  human  as  reliable  (Shattuck  &  Miller,  2006). 

4.  Baseline  Information 

Baseline  data  represents  information  that  is  available,  understood,  and  possesses 
negligible  uncertainty. 
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5.  Distinguishing  the  Definitions 

Crafting  understandable  and  distinguishable  definitions  for  the  categories  of 
uncertainty  was  not  a  trivial  task.  Crafting  suitable  definitions  involved  discussions, 
much  iteration,  and  multiple  pilot  studies  to  truly  capture  the  essence  of  each  category. 
Historical  examples  that  illustrate  the  differences  are  provided  below. 

Ambiguous/missing  information  was  presented  to  French  General  Napoleon 
Bonaparte  at  the  Battle  of  Waterloo.  On  June  18th,  1815,  the  French  army  prepared  to 
destroy  the  Duke  of  Wellington’s  English  force.  Days  earlier,  Napoleon  defeated  the 
Prussian  Annies  at  Ligny.  However,  his  victory  over  the  Prussians  came  with  a  price. 
The  Prussians  headed  east,  away  from  French  scouts.  Without  intelligence  on  important 
information  such  as  location,  heading,  speed  of  advance,  casualties,  or  intentions  of  the 
Prussian  armies,  the  French  general  was  placed  in  a  difficult  and  frustrating  position. 
Napoleon  wanted  to  attack  the  left  flank  of  the  British  forces,  but  the  idea  of  the  Prussian 
Army  somewhere  to  his  right  flank  was  not  only  dangerous  but  unnerving  (Keegan, 
1978).  The  reality  of  the  ensuing  battle  left  the  “genius”  of  Napoleon  marginalized.  The 
ambiguous  nature  of  the  Prussian  whereabouts  led  Napoleon  to  conclude  that  a  frontal 
attack  was  his  best  option  (Keegan,  1978).  Consequently,  he  was  defeated,  and  forced  to 
surrender. 

In  contrast,  on  October  1962,  conflicting  information  was  presented  to  President 
John  F.  Kennedy.  On  October  14,  1962,  U.S.  intelligence  took  photographs  of  Soviet 
Union  nuclear,  ballistic  missiles  in  Cuba.  The  placement  of  nuclear  weapons  just 
hundreds  of  miles  off  the  coast  of  Florida  placed  U.S.  national  security  at  grave  risk. 
President  Kennedy  chose  not  to  inform  the  Soviets  of  his  discovery.  Instead,  the 
President  and  his  advisor  debated  over  alternatives  for  an  appropriate  response.  However, 
on  October  18th,  President  Kennedy  met  with  the  Soviet  Minister  of  Foreign  Affairs, 
Andrei  Gromyko.  When  asked  about  the  nature  of  Soviet  weapons  in  Cuba,  Soviet 
Minister  Gromyko  told  President  Kennedy  that  the  Soviet  Union  weapons  were  of  a 
defensive  nature  (Brugioni,  1991;  Hilsman,  1996).  While  President  Kennedy  most  likely 
immediately  dismissed  Gromyko’s  claim,  the  Soviet  Minister  of  Foreign  Affairs  provided 
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the  President  with  a  classic  example  of  conflicting  infonnation;  either  the  Soviet  Union 
did  or  did  not  have  offensive  weapons  in  Cuba.  Consequently,  President  Kennedy  had 
two  alternatives  that  were  contradictory  in  nature. 

F.  UNCERTAINTY  TAXONOMY 

The  uncertainty  taxonomy  was  developed  to  better  understand  the  factors  that 
create  and  reason  with  uncertainty  in  the  battle  space.  Like  the  DMSC,  the  uncertainty 
taxonomy  examines  the  factors  that  influence  decision  making  under  uncertainty  (see 
Figure  5).  While  human  cognition  is  responsible  for  comprehension  and  projection  of 
uncertainty,  multiple  factors,  including  technological  systems,  communication,  and 
individual  differences  shape  the  conditions  in  which  uncertainty  can  arise.  Consequently, 
the  uncertainty  taxonomy  provides  means  for  classifying  uncertainty  types  and  for 
suggesting  viable  areas  of  research  to  determine  the  impact  of  the  uncertainty  types  on 
performance.  In  order  to  better  explain  the  relevance  of  the  uncertainty  taxonomy,  the 
researchers  divided  into  two  parts:  (1)  infonnation  flow  (IF)  and  (2)  individual  difference 
factors  (IDF). 

Like  the  DSMC,  information  flow  focuses  on  the  projection  of  information  or 
sources  of  information  in  the  battle  space.  This  structure  parallels  McCloskey  and  Klein’s 
(1996)  levels  of  uncertainty.  The  IF  is  not  just  limited  to  humans,  but  also  machines. 
Conversely,  IDFs  focus  solely  on  the  human’s  abilities  and  cognition.  This  coincides  with 
Shattuck  and  Miller’s  (2006)  Lens  A  and  Lens  B  in  the  DMSC.  However,  it  is  important 
to  note,  trust  is  the  interlinking  variable  (unreliable  information)  in  both  the  IF  and  IDF 
branches  (i.e.,  how  well  does  an  individual  trust  one’s  self,  in  addition  to,  trusting  the 
technology?). 

The  ultimate  goal  of  the  uncertainty  taxonomy  is  to  analyze  these  associations 
with  respect  to  their  influence  on  decision  making.  Consequently,  the  IDF  section 
discusses  the  influence  of  experience  and  personality,  and  examines  how  individuals  cope 
with  uncertainty. 
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Figure  5.  Uncertainty  Association  Taxonomy. 

1.  Information  Flow 

IF  is  critical  in  military  operations.  Whether  at  the  team  performance  level  or 
individual  performance  level,  the  efficiency  and  speed  of  IF  can  determine  the  outcome 
of  the  engagement.  Without  adequate  IF,  a  unit  or  commander  may  be  paralyzed  or 
unable  to  make  critical,  time-sensitive  decisions  due  to  the  inconsistency  or  unavailability 
of  information.  Consequently,  the  interactions  of  several  factors  contribute  to  the 
aggregate  effectiveness  of  IF  in  conducting  continuous  operations. 

a.  Information  Flow  Interaction 

The  interaction  of  trust,  data,  sources,  uncertainty,  and  communication  is 
complex.  Instead  of  viewing  these  terms  as  abstract,  independent  ideas,  the  research 
addressed  them  using  a  systems  approach  (i.e.,  rather  than  considering  these  terms 
individually,  the  present  research  addressed  their  interaction).  First,  uncertainty  can  result 
from  the  interaction  of  data  and  communication.  Data  can  be  in  the  form  of 
ambiguous/missing  information,  conflicting  information,  unreliable  infonnation,  or 
baseline  infonnation.  Additionally,  the  communication  of  data  can  be  the  aforementioned 
categories  of  uncertainty.  For  example,  if  an  Officer  of  the  Deck  (OOD)  witnesses  on 
radar  two  contacts  off  the  starboard  bow,  and  a  sailor  looking  at  the  same  input  from  a 
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different  display  reports  three  contacts  off  the  starboard  bow,  then  the  communication  of 
the  data,  not  the  raw  data  itself,  is  uncertain.  Conversely,  if  the  sailor  witnesses  three 
contacts  off  the  starboard  bow  from  a  different  input  source  on  a  different  screen,  then  the 
raw  data,  not  the  communication  of  the  data  are  uncertain. 

Second,  sources  can  be  uncertain  (i.e.,  a  source,  whether  a  system  or 
human  can  be  conflicting,  unreliable,  or  ambiguous/missing).  In  addition  to  being 
uncertain,  sources  can  possess  the  characteristic  of  trustworthiness.  Kramer  (1999) 
defines  trust  as  “a  state  of  perceived  vulnerability  or  risk  that  is  derived  from  individual’s 
uncertainty  regarding  the  motives,  intentions,  and  prospective  actions  of  others  on  whom 
they  depend”  (p.  571).  In  other  words,  trust  is  an  individual’s  acceptance  of  the 
consequences  for  another’s  (man  or  machine)  actions. 

Zand  (1997)  describes  IF  as  a  characteristic  of  trust.  He  discusses  the 
effects  of  “[predisposed]  beliefs”  and  “short-cycle  feedback”  loops  (p.  94).  He  suggests 
that  “people  build  a  data  bank,  gathering  impressions  of  other’s  trustworthiness... these 
impressions  are  short-cycle  feedback  loops;  they  confirm  or  disconfirm”  an  individual’s 
trustworthiness  (p.  94).  Additionally,  Yanik  and  Kleinberger  (2000)  suggest  that  decision 
makers  are  constantly  correlating  the  quality  of  the  advice  given  to  the  individual  who 
gave  them  the  advice. 

Even  in  the  absence  of  feedback  loops,  decision  makers,  as  reflected  in 
their  advice  weighing  policies,  are  aware  of  the  quality  of  the  advice  received  (Yanik  & 
Kleinberger,  2000).  The  researchers  offer  two  suggestions  for  the  phenomena:  (a) 
decision  makers  “relied  on  ranges  of  estimates”  to  determine  accuracy,  and  (b)  decision 
makers  may  have  performed  “plausibility  checks,”  and  “recognized  that  a  particular 
estimate  was  out  of  bounds”  (Yanik  &  Kleinberger,  pg.  270).  Not  only  does  this  suggest 
that  trust  relies  on  the  acceptance  of  the  consequences  for  another’s  actions,  but  it  also 
shows  that  prior  knowledge  or  experience  often  guides  decision  makers. 

2.  Individual  Difference  Factors 

While  IF  is  critical  to  C2  at  the  operational  level,  IDFs  translate  into  individual 
performance.  IDFs  consist  of  several  elements,  including  both  intrinsic  and  extrinsic 
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factors.  Personality  characteristics,  knowledge,  cognitive  processing,  social  interaction, 
training,  and  experience  all  contribute  to  a  decision  maker’s  ability  to  perform  (Miller, 
N.L.,  Crowson,  J.J.,  &  Narkevicius,  J.M.,  2003).  Consequently,  researchers  must 
examine  human  behavior  under  various,  unpredictable  conditions  to  fully  grasp  and 
eventually  predict  human  behavior  (Miller  et  al.,  2003). 

How  an  individual  responds  to  adversity,  risk,  or  uncertainty  may  play  a  vital  role 
in  their  choice  of  alternatives.  This  was  the  key  question  of  the  researchers’  work,  and  for 
the  scope  of  the  study,  they  examined  how  individuals  cope  with  uncertainty  and  how  it 
related  to  their  decision  times.  In  the  present  study,  the  researchers  focused  on  three 
aspects  of  specific  IDFs:  (a)  operational  experience,  (b)  personality  characteristics  (as 
measured  by  NEO  Five  Factor  Indicator  (NEO-FFI)),  and  (c)  the  ability  to  cope  with 
uncertainty  (as  measured  by  Uncertainty  Response  Scale  (URS)). 

a.  Operational  Experience 

An  individual’s  competence  is  derived  from  his  or  her  ability  to  transform 
prior  experience  to  decisive  action.  Klein’s  (1997)  research  using  the  Recognition  Primed 
Decision  (RPD)  model  (Klein,  Calderwood,  &  Clinton-Cirocco,  1985)  supports  this 
claim.  Klein  (1997)  found  that  decision  makers,  in  dynamic  environments,  mostly  rely  on 
past  experience  when  deciding  on  their  course  of  action.  Kaempf,  Wolf,  Thordsen,  and 
Klein  (1992,  as  cited  in  Klein,  1997,  p.  289)  examined  78  cases  of  decision  making  on 
U.S.  Navy  AEGIS  Cruisers,  and  estimated  that  in  78%  of  the  cases  the  “decision  maker 
adopted  the  course  of  action  (COA)  without  any  deliberate  evaluation”  (p.  289).  This 
suggests  that  decision  makers  do  not  weigh  alternatives,  but  rather  choose  based  on  prior 
experience  (Kaempf  et  al.,  1992;  Klein,  1997). 

St.  John,  Callan,  Proctor,  and  Holste  (2000)  used  Tactical  Decision  Games 
(TDG)  to  measure  how  uncertainty  affects  decision  making  among  U.S.  Marines.  The 
researchers  found  that  when  facing  increased  uncertainty,  the  majority  of  less 
experienced  Marine  officers  chose  a  “wait  and  see”  option,  while  the  more  experienced 
officers  did  not  (St.  John  et  al.,  2000).  This  study  coincides  with  the  findings  of  Serfaty, 
MacMillan,  Entin,  and  Entin  (1997),  who  found  that  participants  with  greater  expertise  in 
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a  given  field  provided  a  more  detailed  COA  compared  to  participants  with  less  expertise. 
They  concluded  their  findings  “[were]  consistent  with  the  idea  that  experts  are  able  to 
draw  on  previous  experience  to  generate  a  more  complete  schema  for  the  tactical 
situation”  (Serfaty  et  ah,  p.  242). 

In  this  study,  experience  is  defined  as  operational  experience.  Operational 
experience  is  defined  as  months  in  an  operational  theatre  (e.g.,  Iraq  &  Afghanistan). 
However,  the  researchers  are  confident  that  while  all  the  participants  received  specialized 
training  for  their  occupations,  they  also  received  common  basic  training.  Consequently, 
each  participant  had  enough  training  and  experience  to  be  evaluated  in  the  study. 

b.  NEO-FFI 

Experience  is  the  most  obvious  extrinsic  factor,  but  personality  is  also 
important  when  examining  decision  making  under  uncertainty.  The  NEO  Personality 
Inventory  has  been  the  standard  for  research  in  personality  traits  and  characteristics. 
However,  unlike  the  NEO  Personality  Inventory,  the  NEO-Five  Factor  Indicator  (NEO- 
FFI)  only  addresses  the  five  major  elements  of  personality,  and  not  the  different  facets 
that  accompany  each  (Costa  &  McCrae,  1992,  &  Hannen,  2007).  For  this  study,  the 
NEO-FFI  was  used  to  look  at  the  relationship  between  personality  and  uncertainty.  The 
five  major  elements  of  the  NEO-FFI  are  the  following: 

•  Openness  (O) 

•  Agreeableness  (A) 

•  Conscientious  (C) 

•  Neuroticism  (N) 

•  Extraversion  (E) 

(1)  Openness  (O)  refers  to  individuals  who  are  “open  to 
experience”  and  “curious  about  both  inner  and  outer  worlds,”  (Costa  &  McCrae,  1992,  p. 
15).  Additionally,  individuals  who  score  high  on  O  are  not  usually  conservative  in 
thought,  but  rather  unorthodox  thinkers  with  creative  imaginations  (Costa  &  McCrae, 
1992).  Under  uncertainty,  the  researchers  would  expect  a  decision  maker  to  not 
necessarily  respond  slower  or  faster,  but  rather  adopt  a  clever,  inventive  decision. 
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(2)  Agreeableness  (A)  refers  to  individuals  who  are 
“fundamentally  altruistic”  and  “sympathetic  to  others  and  eager  to  help  them,”  (Costa  & 
McCrae,  1992,  p.  15).  However,  Costa  and  McCrae  (1992)  are  quick  to  note,  that 
agreeableness  is  not  a  preferred  characteristic  in  occupations  where  one’s  own  self- 
interests  may  be  at  risk  (e.g.,  military,  police,  firefighter).  Consequently,  in  a  sample 
pool  involving  military  service  members,  one  would  not  expect  respondents  to  score  high 
on  A.  Under  uncertain  conditions,  the  researchers  would  expect  individuals  who  score 
high  on  A  to  respond  quickly  with  sometimes  irrational,  if  not  brash,  decisions. 

(3)  Conscientiousness  (C)  examines  whether  an  individual  is 
“purposeful,  strong-willed,  and  determined,”  (Costa  &  McCrae,  1992,  p.  16). 
Additionally,  Costa  and  McCrae  (1992)  suggest  conscientiousness  is  somewhat 
analogous  with  character  (i.e.,  individuals  who  possess  high  C  scores  are  usually 
considered  credible  and  in  high  moral  stature).  The  researchers  would  expect  individuals 
who  score  high  on  C  would  not  necessarily  respond  faster  or  slower,  but  be  very  aware  of 
both  ethical  and  moral  elements  of  their  decisions,  not  to  mention,  self-reflective. 

(4)  Neuroticism  (N)  refers  to  the  emotional  stability  of  an 
individual  (Costa  &  McCrae,  1992).  Moreover,  an  individual  who  possess  low  N  scores 
will  usually  be  “calm,  even-tempered,  and  relaxed,”  (Costa  &  McCrae,  1992,  p.  15). 
More  importantly,  they  are  able  to  deal  with  stressful  environments  (Costa  &  McCrae, 
1992).  Military  officers  often  perform  in  complex,  operational  environments,  therefore 
they  would  be  expected  to  score  low  on  N. 

(5)  Extroversion  (E)  examines  whether  or  not  an  individual  is 
“assertive,  talkative,  or  active,”  (Costa  &  McCrae,  1992,  p.  15).  Additionally,  it  explores 
whether  an  individual  socializes  in  larger  groups  or  even  likes  people  in  general  (Costa 
&McCrae,  1992).  Faced  with  uncertainty,  individuals  who  score  high  on  E  would  be 
quick,  even  aggressive,  to  make  a  decision.  However,  it  is  unknown  whether  their 
decision  would  be  ill-thought  or  carefully  calculated. 

In  conclusion,  the  NEO-FFI  may  provide  insight  into  decision 
making  under  uncertainty.  The  different  elements  of  personality  may  lend  to  increased 
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resolve  when  faced  with  uncertainty  in  a  time-sensitive,  stressful  environment.  However, 
we  must  explore  whether  these  characteristics  are  congruent  with  the  methods  or 
techniques  used  to  cope  with  uncertainty. 

c.  Coping  with  Uncertainty 

Although  experience  and  personality  characteristics  may  influence  a 
decision  maker’s  choice  of  alternatives,  perhaps  another  important  measure  may  lie  in 
how  a  commander  responds  emotional  and  cognitively  to  uncertainty.  To  explore  this 
question,  the  Uncertainty  Response  Scale  (URS)  explores  the  effects  of  uncertainty  from 
three,  different  perspectives:  (1)  emotional  uncertainty  (EU),  (2)  desire  for  change  (DFC), 
and  (3)  cognitive  uncertainty  (CU)  (Greco  &  Roger,  2001)  (see  Appendix  A). 

EU  is  defined  as  “the  degree  to  which  an  individual  responds  to 
uncertainty  with  maladaptive  behavior”  (Sutton,  Cosenzo,  &  Pierce,  2004).  Higher  scores 
on  EU  indicate  an  individual  does  not  adapt  well  to  uncertain  situations  (Thomas,  2005). 
Interestingly,  Greco  and  Roger  (2001)  found  a  significant  correlation  between  N  and  EU 
( r  =  0.56,  p  <  0.01).  Additionally,  EU  correlated  significantly  with  Kirton’s  (1981) 
Tolerance  to  Ambiguity  (TOA)  (r  =  0.18,  p  <  0.01).  Consequently,  we  suggest 
individuals  who  score  high  on  EU  and  N  erratic  response  times  in  an  uncertain,  tactical 
environment. 

DFC  is  classified  as  “the  enjoyment  of  uncertainty,  novelty,  and  change,” 
(Greco  &  Roger,  2001,  p.  525).  Higher  scores  on  DFC  indicate  that  an  individual  enjoys 
uncertainty  or  the  unknown  (Thomas,  2005).  Greco  and  Roger  (2001)  found  significant 
correlations  between  sub-elements  of  E  (e.g.,  impulsivity  and  sociability).  The 
researchers  found  a  moderate  association  between  DFC  and  impulsivity  (r  =  0.37,  p  < 
0.01)  and  DFC  and  sociability  ( r  =  0.23 ,  p  <  0.01).  Unfortunately,  these  results  give  no 
insight  into  whether  individuals  with  high  E  scores  will  perfonn  adequately  under 
uncertain  conditions. 

CU  is  described  as  the  “degree  to  which  an  individual  prefers  order, 
planning,  and  structure  in  an  uncertain  environment,”  (Thomas,  2005,  p.  35).  Higher 
scores  on  CU  indicate  that  an  individual  desires  control  or  organization  in  uncertain 

26 


situations  (Thomas,  2005).  Greco  and  Roger  (2001)  found  a  significant  correlation 
between  CU  and  TOA  (r  =  0.37,  p  <  0.01).  Additionally,  the  researchers  discovered  a 
significant  correlation  between  CU  and  N  (r  =  -0.20,  p  <  0.05).  Moreover,  the  E  sub¬ 
element  of  impulsivity  was  significantly  correlated  to  CU  (r  =  -0.25,  p  <  0.01).  As 
previously  mentioned,  military  officers  often  operate  in  highly  structured,  yet  complex, 
environments.  Consequently,  these  results  suggest  officers  scoring  high  in  cognitive 
uncertainty  may  perform  well  when  faced  with  uncertainty. 

These  significant  correlations  provide  evidence  of  the  important 
relationship  between  personality  characteristics  and  an  individual’s  method  for  coping 
with  uncertainty.  The  URS  has  both  high  internal  consistency  reliability  (EU  =  0.91,  CU 
=  0.87,  DFC  =  0.87)  and  test-retest  reliability  (EU  =  0.79,  CU  =  0.80,  DFC  =  0.86),  thus 
procuring  credibility  for  future  research.  More  importantly,  Greco  and  Roger  (2001) 
provide  an  excellent  basis  for  inferring  a  relationship  between  uncertainty  and  personality 
characteristics.  Their  work  has  implications  for  the  present  research  in  tactical  decision¬ 
making  under  uncertainty. 

G.  CRITICAL  DECISION  MAKING  METHOD 

Although  the  decision  maker’s  choice  of  alternative  is  important  to  understanding 
his  or  her  cognitive  reasoning,  it  is  not  the  focal  point  of  this  study.  Instead,  research 
efforts  were  directed  toward  discovering  the  impact  of  uncertainty  on  the  commander’s 
decision.  This  paradigm  shift  is  the  result  of  two  objectives.  First,  the  study  of  uncertainty 
was  undertaken  to  determine  “why  and  how”  uncertainty  affects  the  commander’s  lens. 
Secondly,  the  study  was  focused  in  discovering  where  uncertainty  occurs  in  the  tactical 
environment. 

Until  this  point,  this  literature  review  has  explored  the  “why”  and  “how”  question, 
but  not  adequately  addressed  the  “where.”  Consequently,  before  we  can  develop  a 
methodology  to  test  uncertainty  in  decision  making,  we  must  expose  and  unearth 
uncertainty  in  the  battle  space,  (i.e.,  in  the  ovals  in  the  DMSC)  (Shattuck  and  Miller, 
2003).  In  simpler  tenns,  we  must  find  a  process  to  elicit  uncertain  situations  in  the 
operational  world. 
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Klein  et  al.  (1989)  developed  the  elicitation  process  of  Critical  Decision  Method 
(CDM).  The  techniques  of  examining  cases  based  on  non-routine  events  by  cognitive 
probes  and  maintaining  a  semi-structured  approach  allow  researchers  to  uncover  the 
methods  and  reasoning  behind  an  individual’s  decision.  These  techniques  follow  a  top- 
down  approach  to  detennine  how  and  why  a  decision  maker  chose  an  alternative.  More 
importantly,  as  Klein  et  al  (1989)  points  out,  CDM  provides  a  method  for  evaluating 
performance  to  better  assist  in  identifying  training  requirements.  Consequently,  CDM  is  a 
powerful  tool  in  reconstructing  naturalistic  events  in  the  hopes  of  creating  more  realistic 
and  efficient  models  for  training  simulations. 

While  Klein’s  interest  lies  in  the  decision  maker’s  cognitive  reasoning,  CDM  was 
modified  in  this  study  to  elicit  the  presence  of  uncertainty  in  the  battle  space.  Developed 
by  Dr.  Lawrence  Shattuck,  the  modified  CDM  follows  the  outline  presented  by  Klein  et 
al  (1989),  but  instead  uses  cognitive  probing  techniques  to  unearth  the  presence  (i.e.,  the 
“where”)  of  uncertainty  in  the  participant’s  recollection  of  events  (see  Appendix  B). 
Consequently,  the  decision  becomes  secondary  in  the  process.  This  alteration  allows  the 
researchers  to  categorize  the  witnessed  uncertainty  using  the  definitions  provided  in 
Chapter  II.F.  Moreover,  it  enables  a  clear  examination  of  the  chronological  events  prior 
to  and  after  uncertainty  appears  in  the  equation. 

H.  HYPOTHESES 

This  literature  review  has  uncovered  many  important  questions  in  the  field  of 
tactical  decision-making  under  uncertainty.  The  questions  have  been  narrowed  to  those 
deemed  most  important.  The  alternative  hypotheses  generated  from  those  questions  are 
listed  below. 

•  Hai:  There  is  a  significant  difference  in  the  overall  total  time  with  respect 
to  the  three  categories  of  uncertainty. 

•  Ha2a:  There  is  a  significant,  positive  correlation  between  emotional 
uncertainty  and  overall  total  time  for  ambiguous/missing  information  and 
conflicting  information. 

•  Haib:  There  is  a  significant,  negative  correlation  between  cognitive 
uncertainty  and  overall  total  time  for  ambiguous/missing  information  and 
conflicting  information. 
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•  Ha^:  There  is  a  significant,  negative  correlation  between  operational 
experience  and  overall  total  time  for  ambiguous/missing  infonnation  and 
conflicting  information. 
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III.  METHOD 


Research  began  in  January  2007.  Over  the  past  18  months  three  small  scale  pilot 
studies  were  conducted.  The  results  shaped  the  methods  used  in  the  main  study.  This 
section  contains  short,  concise  descriptions  of  the  three  pilot  studies,  in  addition  to, 
lessons  derived  from  their  results.  Finally,  the  main  study  method  will  be  outlined. 

A.  PILOT  STUDIES 

This  section  addresses  the  pilot  studies  conducted  by  the  researchers.  Each  pilot 
study  focused  on  the  research  questions  outlined  in  Chapter  II. C.  It  is  important  to  note, 
the  latter  pilot  studies  built  off  the  successes  and  failures  of  the  preceding  studies. 
Consequently,  they  will  be  discussed  in  chronological  order. 

1.  Pilot  #1  -  Ambiguous  Words 

In  the  first  pilot  study,  a  research  team,  consisting  of  Naval  Postgraduate  School 
(NPS)  students,  attempted  to:  (1)  operationally  define  ambiguity,  and  (2)  design  a  study 
to  test  its  effects  on  decision-making  with  respect  to  the  time  it  takes  to  make  a  decision. 
After  a  review  of  the  literature,  the  team  began  developing  models  to  understand 
uncertainty.  The  team  considered  the  research  of  McCloskey  and  Klein  (1996)  as  the 
most  robust  definition  of  ambiguity  and  uncertainty.  Their  work  provided  the  basis  to 
design  an  experiment  examining  uncertainty  at  the  fundamental  level. 

a.  Participants 

Thirty-two  NPS  students  (30  male,  2  female)  volunteered  to  participate  in 
the  initial  pilot  study.  All  participants  were  commissioned  officers,  who  were  in  the 
process  of  earning  their  master’s  degree.  Of  the  group,  twenty-nine  participants  had  20/20 
vision,  while  only  three  reported  being  colorblind.  The  colorblind  participants  were 
omitted  from  the  results.  No  compensation  was  given  for  their  participation. 
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b. 


Method 


The  participants  were  shown  numerous  Microsoft  PowerPoint  slides  with 
a  word  in  pink  letters  (see  Appendix  C).  The  words  were  drawn  from  the  American 
National  Standards  Institute  (ANSI)  Standard  S3. 2- 1989,  Method  for  Measuring  the 
Intelligibility  of  Speech  over  Communications  System  (2005).  Transposed  over  the  pink 
letters  was  a  rectangular  pink  square,  which  was  adjusted  for  resolution  to  create  different 
severities  of  ambiguity  (high,  medium,  low).  The  participants  were  given  four  choices 
and  asked  to  match  the  word  shown  on  the  slide  to  a  word  on  their  answer  sheet. 
Confidence  was  measured  by  survey  using  a  Likert  scale.  Time  to  decision  (time  to 
decision)  was  measured  by  an  electronic  timer  embedded  in  the  software.  Participant 
accuracy  was  detennined  by  grading  the  correctness  of  their  responses. 

c.  Results 

The  researchers  analyzed  three  separate  aspects  of  the  data  using  a  within 
subjects  design.  First,  using  ANOVA,  the  researchers  found  a  significant  difference 
between  the  three  severities  of  ambiguity  (high,  medium,  low)  with  respect  to  time  to 
decision  (F  (2,  320)  =  76.20,  p  <  0.001).  Next,  the  researchers  used  non-parametric 
measures  (contingency  analysis)  to  determine  the  relationship  between  confidence  and 
ambiguity  severity  level.  As  expected,  they  found  a  higher  percentage  of  “less  confident” 
participant  responses  when  presented  with  the  “high”  severity  of  ambiguity  compared  to 
medium  and  low  severities.  Finally,  the  researchers  analyzed  the  accuracy  rates  of  the 
three  severities  of  uncertainty.  They  discovered  the  following:  High  =  85.6%,  Medium  = 
96.9%,  Low  =  99.7%. 

d.  Discussion 

Overall,  the  study  provided  an  initial  approach  to  the  topic  of  uncertainty 
and  ambiguity.  The  results  suggest  that  ambiguity  severity  affects  response  time. 
Additionally,  the  results  infer  that  accuracy  and  confidence  are  degraded  by  the  presence 
of  ambiguity.  Unfortunately,  the  study  did  not  address  whether  ambiguity  and  uncertainty 
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were,  in  fact,  synonymous  tenns.  If  ambiguity  and  uncertainty  are  not  synonymous,  then 
how  are  they  different  (or  differentiated)?  Is  one  subsumed  by  the  other?  Are  there  other 
categories  that  are  separate  but  related? 

While  these  questions  were  important  to  answer,  another  need  slowly 
emerged  from  the  study.  When  the  results  from  the  first  pilot  were  presented  to  military 
officers  and  civilian  officials,  they  questioned  the  relevance  and  realism  of  the  stimuli 
(i.e.,  how  does  the  study  link  to  the  ambiguity  seen  by  commanders  on  the  battlefield?). 
Consequently,  the  researchers  returned  to  the  NDM  literature  to  find  examples  (solutions) 
of  real-world,  operational  environments.  Additionally,  the  results  and  conclusions  from 
the  first  pilot  study  moved  the  researchers  to  develop  two  distinct  goals  for  the 
subsequent  pilot  study:  (1)  to  develop  a  better  operational  definition  of  uncertainty,  and 
(2)  to  design  a  study  that  incorporates  a  relevant  military  setting. 

2.  Pilot  #2  -  Operational  Intelligence 

For  the  second  pilot  study,  the  research  team  expanded  the  scope  of  research. 
McCloskey  and  Klein  (1996)  suggest  that  four  categories  of  uncertainty  in  the  realm  of 
uncertainty.  They  described  them  as  follows:  (1)  missing  information,  (2)  complex 
information,  (3)  ambiguous/conflicting,  and  (4)  unreliable  information  (McCloskey  and 
Klein,  1996). 

However,  the  researchers’  experience  in  tactical  environments  led  them  to  believe 
ambiguous/conflicting  were  two  separate  categories.  Additionally,  the  idea  of  complex 
information  seemed  too  abstract  to  operationally  define  given  a  relatively  short  timeline. 
Moreover,  complexity  seemed  inherent  in  uncertainty  (i.e.,  complexity  was  proportional 
to  uncertainty).  Consequently,  the  research  team  opted  to  evaluate  three  categories:  (1) 
missing  information,  (2)  conflicting  infonnation,  and  (3)  ambiguous  information. 

a.  Participants 

Thirty-four  participants  from  NPS,  which  included  14  naval  officers,  5 
Marine  Corps  officers,  6  anny  officers,  5  foreign  military  officer,  and  4  civilians 
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participated  in  the  second  pilot  study.  They  ranged  from  two  to  twenty  years  of  military 
service  and  were  not  compensated  for  their  participation. 

b.  Method 

The  participants  were  placed  in  an  operational  situation.  They  were 
briefed  on  a  hostage  crisis  involving  several  senior  U.S.  State  Department  officials, 
including  the  Secretary  of  State.  The  experimenters  explained  that  intelligence  agents  had 
placed  three  GPS  transmitters  on  key  hostage  personnel  in  preparation  for  a  hostage 
rescue  mission.  The  three  GPS  transmitters  each  displayed  a  single  shape  (circle,  triangle, 
square)  on  a  3  x  3  targeting  grid  (see  Figure  6). 


Figure  6.  Pilot  Study  #2  Situational  Display. 

Each  participant  was  shown  20  slides  that  contained  the  shapes  under  the 
four  categories  of  uncertainty  (conflicting  infonnation,  baseline  infonnation,  ambiguous 
information,  missing  information)  (see  Appendix  D).  Each  slide  asked  a  question  relating 
to  the  location  of  a  shape.  Each  scenario  was  displayed  for  a  total  of  twenty  seconds,  and 
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the  question  appeared  with  ten  seconds  remaining.  The  participant  was  then  required  to 
analyze  the  scenario  and  answer  the  question  to  the  best  of  his  or  her  ability.  A  ten  second 
interstitial  slide  was  displayed  to  allow  the  individual  to  indicate  his  or  her  certainty 
response  level  on  a  corresponding  Likert  scale.  The  Likert  Scale  ranged  1  to  9  with  1 
corresponding  to  “not  at  all  certain”  and  9  corresponding  to  “100%  certain.” 

c.  Results 

The  researchers  collected  three  sets  of  data:  (1)  certainty  response  level 
(Likert  Scale),  (2)  grid  location  choice,  and  (3)  NEO-FFI.  Non-parametric  statistics  were 
used  to  analyze  the  certainty  response  levels,  while  Pearson  correlation  coefficient  was 
used  for  the  NEO-FFI  and  response  time  data.  The  researchers  found  a  significant 
difference  between  the  four  categories  of  uncertainty  (see  Table  1),  but  did  not  find  a 
significant  correlation  to  any  of  the  five  factors  of  the  NEO-FFI. 


Miss  -  Base 

Con  -  Base 

Amb  -  Base 

Con  -  Miss 

Amb  -  Miss 

Amb  -  Con 

z 

-5.388 

-4.773 

-5.127 

-5.127 

-4.874 

-3.359 

Asymp.  Sig. 

(2-tailed) 

.000 

.000 

.000 

.000 

.000 

.001 

Table  1.  Sign  Test  for  Categories  of  Uncertainty. 


The  study  suggests  that  individuals  differ  in  their  degree  of  certainty  with 
respect  to  the  category  of  uncertainty.  Individuals  appear  to  be  less  certain  when  faced 
with  ambiguous  information  than  with  conflicting  information  (i.e.,  individuals  would 
rather  have  a  choice  between  two  alternatives  vice  a  choice  amongst  many  or  unknown 
alternatives). 


d.  Discussion 

The  results  from  Pilot  Study  #1  and  Pilot  Study  #2  produced  another 
important  question.  If  the  categories  of  uncertainty  contribute  to  significant  differences  in 
certainty  response  levels  and  if  different  severities  for  ambiguity  lead  to  significant 
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differences  in  time  (see  Chapter  III.A.l),  then  will  different  categories  of  uncertainty 
cause  significant  differences  in  time?  To  answer  this  question  the  research  team 
developed  realistic  tactical  vignettes  for  each  category  of  uncertainty. 

3.  Pilot  #3  -  Tactical  Decision  Environments 
a.  FT  Leavenworth,  KS 

(1)  Participants.  Interviews  were  conducted  at  Command 
General  Staff  College  (CGSC),  FT  Leavenworth,  KS,  using  the  modified  version  of 
Klein’s  (1996)  CDM  (see  Chapter  I.G.).  The  participants  consisted  of  20  U.S.  Army 
officers  (2  females  and  18  males  with  a  mean  age  of  36.9  years).  Nineteen  participants 
held  the  rank  of  0-4  or  Major,  while  one  participant  held  the  rank  of  0-3  or  Captain.  The 
participants  had  an  average  of  16.0  years  of  military  service  and  13.1  years  of 
commissioned  service.  These  averages  suggest  many  participants  served  in  the  enlisted 
ranks  prior  to  earning  a  commission.  Additionally,  the  participants  had  an  average  of  15.5 
months  of  deployment  time  post  September  11th,  2001  (i.e.,  the  participants  had  a 
creditable  amount  of  operational  experience  in  the  War  on  Terror) 

(2)  Method.  To  begin  the  knowledge  elicitation,  the 
researchers  used  the  following  paragraph  as  their  primary  inquiry: 

Recall  a  tactical  situation  during  a  recent  deployment  to  a  combat  zone  in 
which  you  were  confronted  with  uncertainty  and  had  to  make  a  decision. 
Uncertainty  can  be  described  simply  as  a  sense  of  doubt  that  blocks  or 
delays  action.  There  are  no  right  or  wrong  answers  and  we  are  in  no  way 
evaluating  your  performance  or  the  decisions  you  made. 

Each  participant  was  asked  to  provide  two  situations  or  scenarios. 
Additional  follow-on  questions  used  for  data  elicitation  can  be  found  in  Appendix  B. 
These  questions  were  used  to  explore  the  uncertainty  in  the  described  situation. 

The  researchers  analyzed  the  scenarios  described  by  the 
participants.  The  majority  of  the  incidents  of  uncertainty  recalled  were  categorized  as 
ambiguous.  Two  scenarios  were  selected  and  modified  for  the  data  collection  at  NPS. 
The  researchers  created  four  versions  of  each  scenario  (i.e.,  conflicting,  missing, 

ambiguous,  and  baseline)  for  a  total  of  eight  situations. 
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b.  Naval  Postgraduate  School 

(1)  Participants.  The  participants  at  NPS  consisted  of  17 
officers,  including  one  female  and  sixteen  males  with  an  average  age  =32.5  years.  Nine 
participants  were  U.S.  Navy  officers,  three  were  U.S.  Army  officers,  one  was  a  U.S.  Air 
Force  officer,  one  was  a  Canadian  Air  Force  officer,  and  three  participants  were  Turkish 
Air  Force  officers.  Six  participants  held  the  rank  of  0-4,  10  held  the  rank  of  0-3,  and  one 
participant  held  the  rank  of  0-2.  None  of  the  participants  were  compensated. 

(2)  Method.  Each  participant  saw  a  total  of  eight  slides.  They 
were  given  two  pieces  of  information:  (1)  operational  order  (OPORD)  information  and 
(2)  additional  infonnation  (AI).  The  OPORD  provided  data  on  the  situation,  enemy 
forces,  friendly  units,  terrain,  and  tactical  situation.  The  additional  data  provided  real  time 
intelligence,  including  enemy  movement,  enemy  killed,  and  situational  reports 
(SITREPS). 

The  OPORD  first  appeared  followed  by  the  AI.  All  the  infonnation 
was  present  within  24  seconds  of  scenario  commencement;  thus,  the  participant  had 
direct  access  to  all  information  available  to  make  a  decision.  Once  a  decision  was  made, 
the  participant  pressed  the  space  bar,  which  recorded  the  total  time  from  beginning  to 
end.  However,  the  total  time  was  then  subtracted  by  24  seconds  to  produce  time  to 
decision. 

After  the  participant  pressed  the  space  bar,  the  interviewer,  located 
in  the  same  room,  asked  the  participant  to:  (1)  provide  a  brief  description  of  the  decision, 
and  (2)  list  any  contributing  factors  to  the  decision?  The  researchers  took  written  notes  of 
the  participant’s  responses.  Digital  voice  recorders  (DVRs)  recorded  each  session.  The 
quality  and  accuracy  of  their  answers  were  not  analyzed. 

(3)  Results.  The  researchers  analyzed  the  difference  scores  of 
time  to  decision  for  each  category  of  uncertainty.  The  distributions  of  the  difference 
scores  of  time  to  decision  for  each  category  of  uncertainty  violated  the  nonnality 
assumption,  thus  non-parametric  tests  were  conducted  for  analysis. 
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First,  the  researchers  conducted  an  analysis  to  detennine  whether 
or  not  a  learning  effect  occurred  between  Situation  1  and  Situation  2.  Due  to  the  small 
sample  size,  the  researchers  set  alpha  (a)  at  0.10.  They  discovered  a  significant  learning 
effect  between  Situation  1  and  Situation  2  (p  =  0.09).  Consequently,  the  researcher  did 
not  analyze  Situation  one  data.  Instead,  they  analyzed  Situation  two  data  by  taking  the 
difference  scores  from  each  category  of  uncertainty  and  subtracting  it  from  the  baseline 
for  each  participant  (e.g.,  ambiguous  information  time  to  decision  -  baseline  infonnation 
time  to  decision  =  delta  (ambiguous  information,  baseline  infonnation)). 

The  researchers  found  a  significant  difference  between  the 
difference  scores  for  ambiguous  information  compared  to  conflicting  information  (p  = 
0.097).  Additionally,  they  discovered  a  significant  difference  for  conflicting  information 
compared  to  baseline  {p  =  0.073).  The  researchers  did  not  find  a  significant  difference 
between  missing  and  ambiguous  information  (p  =  0.52). 

(4)  Discussion.  The  pilot  study  identified  issues  that  needed 
reconciliation  prior  the  final  empirical  study.  First,  the  learning  effect  discovered  between 
Situation  1  and  Situation  2  revealed  the  need  for  several  different,  independent  situations 
along  with  randomization.  Consequently,  the  researchers  incorporated  this  approach  into 
their  final  study.  Secondly,  although  “jointness”  is  often  emphasized  in  the  conduct  of 
U.S.  military  operations,  many  of  the  participants  were  naval  officers  vice  army  officers, 
thus  some  tenns  were  unfamiliar  to  the  Naval  officers.  Therefore,  the  researchers 
determined  that  they  needed  a  sample  group  that  shared  a  common  language  and 
experience.  Additionally,  a  list  of  acronyms  was  necessary  to  properly  identify  tenns. 
Thirdly,  the  OPORD  and  AI  was  found  to  be  too  broad  and  generalized,  thus  being 
interpreted  as  “not  real  enough.”  The  researchers  enlisted  the  help  of  a  group  of  active 
duty  or  retired  Army  personnel  to  craft  better,  more  detailed  vignettes. 

Finally,  and  most  importantly,  the  data  suggested  that  ambiguous 
information  and  missing  information  were  statistically  the  same,  thus  the  researchers  (as 
discussed  in  Chapter  II. E)  crafted  new  definitions  and  categories  of  uncertainty.  These 
definitions  became  the  final  definitions  for  the  CGSC  empirical  study. 


38 


B.  CGSC  EMPIRICAL  STUDY 

1.  Independent  Variables 

•  Category  of  Uncertainty  -  ambiguous/missing  information,  conflicting 

information,  baseline  information. 

•  Block  Number  (Randomization)  -  Block  1,  Block  2,  and  Block  3. 

•  Difficulty  -  mean  difficulty  rating  calculated  through  inner  rater  reliability 
testing. 

2.  Dependent  Variables 

•  Overall  Total  Time  (OTT)  -  the  elapsed  time  from  commencement  of 

scenario  to  end  of  scenario. 

•  Situation  time  (ST)  -  the  elapsed  time  from  the  commencement  of  the 
scenario  to  the  when  the  participant  requests  the  additional  information. 

•  Additional  time  (AT)  -  the  elapsed  time  from  the  time  the  participant 
requests  additional  information  to  when  they  finish  reading  the  additional 
information. 

•  Situation/Additional  Total  Time  (SATT)  -  the  elapsed  time  from  the 
commencement  of  the  scenario  to  when  the  participant  finishes  reading  the 
additional  information. 

•  Time  to  Decision  (TTD)  -  the  elapsed  time  from  when  the  participant 
finishes  reading  the  additional  information  to  when  they  inform  the 
researcher  they  have  made  a  decision. 

•  Time  to  Decision/Additional  Total  Time  (TTDAT)  -  the  elapsed  time 
from  when  the  participant  requests  additional  information  to  when  they 
inform  the  researcher  they  have  made  a  decision. 

•  Uncertainty  Response  Scale  Scores  -  the  scores  for  the  three  categories  of 
the  URS  (emotional  uncertainty,  cognitive  uncertainty,  desire  for  change 
uncertainty). 

Overall  total  time,  cognitive  uncertainty,  and  emotion  uncertainty  scores  are  the 
primary  metrics  of  the  study.  The  other  variables  are  secondary  metrics  used  to  discover 
and  explain  differences  in  the  outlined  hypotheses. 

3.  Participants 

The  participants  consisted  of  28  students  attending  the  CGSC,  FT  Leavenworth, 
KS.  There  were  28  males  with  an  average  age  of  37  years  and  an  average  commissioned 
service  time  of  13  years.  Twenty-seven  participants  held  the  rank  of  Major  (0-4),  and  one 
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participant  held  the  rank  of  Lieutenant  Colonel  (0-5).  With  respect  to  their  branch  type, 
14  served  in  Combat  Arms  (CA),  9  served  in  Combat  Support  (CS),  and  5  served  in 
Combat  Service  Support  (CSS). 

Participants  were  treated  in  accordance  with  the  “Ethical  Principles  of 
Psychologists  and  Code  of  Conduct”  (American  Psychological  Association,  1992).  The 
NPS  Institutional  Review  Board  (IRB)  and  CGSC  Quality  Assurance  Office  Survey 
Control  approved  the  research  methods  used  for  the  experiment;  all  participants  signed 
voluntary  consent  forms  and  audio/video  consent  forms  prior  to  participation. 

4.  Materials  and  Apparatus 
a.  Demographic  Survey 

The  demographic  survey  asked  for  information  on  age,  military  service, 
commissioned  service,  branch,  and  operation  deployment  time.  Because  the  study 
focused  entirely  on  U.S.  Army  officers,  the  questions  were  limited  to  the  specified 
service. 


b.  NPS  Decision-Making  Study 

From  the  lessons  learned  in  the  pilot  study  and  the  situations  depicted  by 
the  CDM  interviews,  ten  vignettes  were  crafted  by  members  of  U.S.  Army  Training  and 
Doctrine  Command,  FT  Leavenworth  (TRAC-FLVN),  and  U.S.  Anny  Training  and 
Doctrine  Command,  Monterey  (TRAC-MTY).  The  ten  vignettes  consisted  of  three 
ambiguous/missing  information  vignettes,  three  conflicting  information  vignettes,  and 
four  baseline  information  vignettes  (see  Appendix  E). 

The  vignettes  were  given  to  retired  U.S.  Army  service  members  working 
at  TRAC-FLVN  for  reliability  screening.  Each  evaluator  was  given  the  definitions  listed 
in  Chapter  II. E,  and  the  ten  vignettes.  The  evaluators  were  instructed  to  answer  three 
questions:  (1)  what  category  best  describes  the  uncertainty  in  scenario,  (2)  how  difficult 
would  it  be  to  make  a  decision  in  this  scenario,  and  (3)  how  difficult  was  it  to  classify  the 
category  of  uncertainty? 
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After  several  iterations,  the  researchers  achieved  90%  reliability  for  9  out 
of  the  10  vignettes.  Consequently,  the  researchers  used  those  9  vignettes  for  the  study 
(e.g.,  (3)  ambiguous/missing  vignettes,  (3)  conflicting  vignettes,  and  (3)  baseline 
vignettes).  Additionally,  the  researchers  achieved  high  inter-rate  reliability  for  the  level 
of  difficulty  (k  =  0.81).  The  level  of  difficulty  was  different  for  each  scenario  (see  Table 
2),  and  discussed  and  analyzed  in  Chapter  IV.  The  vignettes  were  counter-balanced  and 
given  randomly  to  the  participants. 


Category 

Difficulty  Mean 

Ambiguous/Missing  1 

3.4 

Ambiguous/Missing  2 

4.7 

Ambiguous/Missing  3 

3.6 

Baseline  1 

2.5 

Baseline  2 

2.3 

Baseline  3 

1.4 

Conflicting  1 

4.1 

Conflicting  2 

2.9 

Conflicting  3 

4.2 

Table  2.  Scenario  Difficulty  Means  for  the  Categories  of  Uncertainty. 

Each  slide  contained  two  segments  of  information:  (1)  situation 
information,  and  (2)  additional  information.  The  situation  information  contained 
information  on  mission,  enemy  size,  friend  forces,  and  environment.  The  additional  data 
provided  real-time,  battle  space  intelligence.  For  all  vignettes,  the  situation  infonnation 
and  additional  information  possessed  approximately  the  same  amount  of  data  (see 
Appendix  E). 

The  researchers  collected  data  on  five  different  time  intervals:  (1)  overall 
total  time,  (2)  time  to  decision,  (3)  situation  time,  (4)  additional  time,  (5)  situation 
time/additional  time,  and  (6)  time  to  decision/additional  time.  Further  explanation  of 
these  times  is  discussed  in  Chapter  III.B.2. 

Stopwatches  were  used  as  the  primary  means  of  recording  the  times.  The 
secondary  means  were  DVRs.  Training  “clickers”  were  used  to  make  distinguishable 
sounds  for  data  analysis.  The  vignettes  were  presented  using  Microsoft  PowerPoint,  and 
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displayed  on  a  52  inch  flat,  plasma  screen  television.  The  study  was  conducted  in  Lewis 
and  Clark  Memorial  Hall,  2nd  floor,  at  CGSC  FT  Leavenworth,  KS  (see  Figure  7). 


Figure  7.  Photograph  of  CGCS  Study. 

c.  Uncertainty  Response  Scale  (URS)  Survey 

As  described  in  Chapter  II.F.2,  the  URS  explored  the  various  ways 
individuals  cope  with  uncertainty.  The  URS  data  were  used  to  help  explain  differences  in 
performance  across  participants. 


d.  Operational  Experience 

Operational  experience  was  collected  by  the  aforementioned  surveys. 
Operational  experienced  was  measured  by  months  in  Afghanistan,  months  in  Iraq,  and 
total  number  of  months  in  Iraq  and  Afghanistan.  Participants  with  no  operational 
experience  in  both  theatres  were  omitted  from  the  study. 
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5.  Design  and  Procedure 

The  study  was  a  within  subjects  design.  Upon  entering  the  room,  participants 
signed  an  Informed  Consent  Form  and  an  Audio/Visual  Consent  Form.  The  participants 
were  seated  at  a  desk  approximately  five  feet  from  the  television  screen.  Next,  they  were 
asked  to  read  an  instruction  sheet  aloud  (see  Appendix  F).  Then,  the  participants  were 
shown  a  slide  with  various  assumptions  about  the  tactical  environment  (see  Figure  8). 
Subsequently,  the  participants  completed  an  example  slide.  Lastly,  the  participants  were 
shown  9  slides  and  times  were  recorded  as  mentioned  previously.  Each  scenario 
proceeded  in  the  following  manner: 


***IMPORTANT*** 


•  You  will  given  a  total  of  10  scenarios;  1 
example  scenario  and  9  evaluated  scenarios. 

•  For  al[  scenarios,  the  following  is  true: 

1)  Your  units  are  properly  equipped  and  capable  of 
accomplishing  the  mission. 

2)  If  you  are  an  advisor,  consider  your  advice  equal  to 
decision. 

3)  Your  command  climate  allows  you  to  make 
decisions  at  your  level. 

4)  The  scenarios  ere  not  connected  in  anyway.  They 
are  independent,  so  consider  every  decision  made 
in  each  scenario  as  separate  and  distinct  from  any 
other  scenario. 


Figure  8.  Tactical  Decision-Making  Assumptions. 

First,  the  participants  were  shown  the  situation  information  and  the  time  started. 
The  participants  read  the  situation  data  aloud.  When  they  were  ready  to  receive  the 
additional  data,  they  snapped  the  training  clicker.  The  training  clicker  was  used  to 
facilitate  the  analysis  of  the  DVR  data  after  the  experiment  (i.e.,  the  training  clicker 
created  a  sound  with  greater  amplitude  than  a  normal  speaking  voice,  thus  making  it 
easier  to  identify  the  various  dependent  variables  associated  with  time). 
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When  the  participants  snapped  their  clicker  for  the  additional  data  two  events 
occurred:  (1)  the  experimenter  logged  their  time  (situation  time)  and  (2)  the  additional 
data  was  presented  to  the  participants.  However,  it  is  important  to  note,  that  the  situation 
data  remained  displayed,  so  the  participant  had  access  to  all  presented  information. 


Second,  when  the  additional  data  appeared,  the  participant  read  the  additional  data 
aloud.  When  the  participants  finished  reading  the  last  word  of  the  additional  data,  the 
experimenter  logged  their  time. 


Third,  when  the  participants  were  ready  to  make  a  decision,  they  were  asked  to 
snap  their  training  clicker.  Upon  snapping  their  training  clicker,  the  experimenter  logged 
their  time.  Thus,  a  total  of  three  times  were  logged,  and  used  to  calculate  the  various 
times  noted  (see  Figure  9). 


Participant  snaps  clicker  and 
receives  additional 
information. 


Participant  finishes  reading 
additional  information. 


Time  =  0.0  seconds 


Participant  snaps  clicker  to 
indicate  they  have  made  a 
decision. 


Situation  Time/Additional 
Time  (SATT) 


Additional  Time/Time  to  / 
Decision  (TTDAT) 


Figure  9.  Tactical-Decision-Making  Assumptions. 

Finally,  the  participant  was  asked  to  answer  two  questions:  (1)  what  is  your 
decision,  and  (2)  how  did  you  arrive  at  your  decision?  The  participants’  answers  were 
recorded  on  DVRs,  and  later  transcribed  for  evaluation. 
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Upon  completion  of  the  NPS  tactical  decision-making  study,  the  participants 
completed  two  surveys.  First,  they  completed  the  URS.  The  title  of  the  survey  was  not 
listed  on  the  paper.  Next,  the  participant  completed  the  demographics  survey  explained  in 
Chapter  II.C.4.  To  ensure  all  documents  and  exercises  were  completed,  the  experimenter 
completed  and  signed  an  interviewer  checklist. 
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IV.  RESULTS 


The  results  consist  of  four  parts:  (a)  summary  statistics  for  categorical  uncertainty, 
(b)  summary  statistics  for  URS,  (c)  summary  statistics  for  operational  experience,  and  (d) 
statistical  analysis  for  the  associations  between  categorical  uncertainty,  URS,  and 
operational  experience.  Descriptive  statistics  are  provided  in  parts  a,  b,  and  c.  Inferential 
statistics  are  used  in  all  parts  to  analyze  differences  across  the  categories  of  uncertainty 
with  respect  to  time,  along  with  appropriate  correlations  between  variables  were 
necessary.  Due  to  discovered  differences  in  the  recorded  times,  between  the  stopwatches 
and  DVRs,  the  researchers  opted  to  use  the  DVR  times  for  statistic  analysis. 

A.  SUMMARY  STATISTICS  FOR  CATEGORICAL  UNCERTAINTY 

For  description  purposes,  additional  time,  situation  time,  and  time  to  decision  will 
be  referred  to  as  the  adjoined  times.  Adjoined  emphasizes  that  these  times  follow  a 
chronological  order  (i.e.,  they  are  linked  by  one  ending  and  the  next  beginning). 
Consequently,  situational  time  is  followed  by  additional  time  which  is  followed  by  time 
to  decision.  Additionally,  situation  time/additional  time  and  time  to  decision/additional 
time  will  be  referred  to  as  the  interrelated  times.  Interrelated  emphasizes  that  these  times 
overlap,  and  that  they  both  use  additional  time  in  their  calculation.  Moreover,  overall 
total  time  will  be  referred  to  as  the  decision  time.  The  decision  time  is  the  total 
summation  of  the  adjoined  times.  This  terminology  was  developed  to  simplify  reporting 
the  results.  Overall  total  time  serves  at  the  variable  of  primary  analysis.  The  interrelated 
and  adjoined  times  are  supplementary  analysis  that  will  be  used  to  better  explain  the 
differences  in  overall  total  time. 

1.  Descriptive  Statistics  for  Categorical  Uncertainty 

For  the  adjoined  times,  ambiguous/missing  information  had  the  largest  mean  for 
situation  time  and  time  to  decision,  while  conflicting  infonnation  had  the  largest  mean  for 
additional  time  (see  Table  3).  Additionally,  ambiguous/missing  had  the  largest  standard 
deviation  (SD)  across  all  the  adjoined  times.  For  overall  total  time,  ambiguous/missing 


had  the  largest  mean  along  with  the  largest  SD. 
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Ambiguous/Missing 

Baseline 

Conflicting 

ST 

51.7  (SD  =  12.2) 

36  (SD  =  8.6) 

35.4  (SD  =  8.3) 

AT 

25.2  (SD  =  2.7) 

24.2  (SD  =  3.5) 

27.1  (SD  =  3.7) 

SATT 

76.9  (SD  =  13.3) 

60.2  (SD  =  10.8) 

62.5  (SD  =  9.9) 

TDD  AT 

43.1  (SD  =  16.7) 

38.5  (SD  =  15.6) 

41.2  (SD  =  11 .7) 

TTD 

18  (SD  =  15.9) 

14.3  (SD  =  13.5) 

14.1  (SD  =  9.4) 

OTT 

94.8  (SD  =25.6) 

74.8  (SD  =  21 .6) 

76.4  (SD  =  17.9) 

Table  3.  Categorical  and  Time  Segment  Means  and  Standard  Deviations  for  CGSC 

Study  (seconds). 

2.  Inferential  Statistics  for  Categorical  Uncertainty 

The  data  violated  the  nonnality  assumption,  thus  non-parametric  analysis  was 
conducted  to  compare  the  categories  of  uncertainty  with  respect  to  time.  To  evaluate  the 
times  for  each  participant,  the  researchers  calculated  the  means  for  each  participant  in 
each  category  of  uncertainty.  Each  participant  had  mean  times  for  each  category  of 
uncertainty  for  each  segment  of  time.  The  researchers  used  the  times  and  difficulty  scores 
to  determine  whether  difficulty  was  a  significant  factor  in  determining  response  time. 


a.  Decision  Time 

For  overall  total  time,  the  researchers  discovered  significant  differences 
amongst  two  of  the  categorical  pairs  of  uncertainty:  (1)  ambiguous/missing  information 
was  significantly  greater  than  conflicting  information  (p  <  0.01),  and  (2) 

ambiguous/missing  information  was  significantly  greater  than  baseline  information  (p  < 
0.01).  Conversely,  the  researchers  did  not  find  a  significant  difference  between 
conflicting  information  and  baseline  information  (p  =  0.06)  (see  Table  4). 


Mean(OTT ,  Baseline)- 
Mean(OTT, 
Ambiguous/Missing) 

Mean(OTT,  Conflicting)-Mean(OTT, 
Ambi  guou  s/Missing) 

Mean(OTT,  Conflicting)- 
Mean(OTT,  Baseline) 

Test  Statistic 

-181 

-184 

80 

Prob  >  |z| 

<.0001 

<0001 

0.0674 

Table  4.  Wilcoxon  Sign-Rank  Analysis  for  Overall  Total  Time. 
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The  researchers  found  a  significant,  positive  correlation  between  all  the 
categories  of  uncertainty  for  overall  total  time:  (1)  ambiguous/missing  information  and 
conflicting  information  (r  =  0.87,  p  <  0.01),  (2)  ambiguous/missing  information  and 
baseline  information  (r  =  0.84,/?  <  0.01),  and  (3)  conflicting  and  baseline  information  (r 
=  0.8,/?  <  0.01)  (see  Figure  10).  Hence,  if  a  participant  responded  quickly  to  one  category 
of  uncertainty,  he/she  would  respond  quickly  to  the  other  categories  of  uncertainty. 


ID# 

Y 

-*■  Mean  (OTT,  A  mbiguous/Missing) 

n  ^“Mean(OTT,  Baseline) 

^  Mean  (OTT,  Conflicting) 

Figure  10.  Participant  Mean  Overall  Total  Time  for  the  Categories  of  Uncertainty. 

Regression  analysis  revealed  that  scenario  difficulty  was  a  significant 
predictor  for  ambiguous/missing  overall  total  time  (p  <  0.01),  but  not  a  significant 
predictor  for  conflicting  overall  total  time  (p  =  0.99)  and  baseline  overall  total  time  (/?  = 
0.95)  (see  Figure  1 1). 
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Figure  11.  Regression  Plot  for  Inter-Rater  Reliability  Difficulty  and  Overall  Total  Time. 
b.  Adjoined  Times 

For  situation  time  ambiguous/missing  information  was  significantly 
greater  than  conflicting  information  (p  <  0.01),  and  significantly  greater  than  baseline 
information  (p  <  0.01).  However,  there  was  no  significant  difference  between  conflicting 
information  and  baseline  information  (p  =  0.45).  Furthennore,  scenario  difficulty  was  a 
significant  predictor  for  ambiguous/missing  situation  time  (p  <  .01),  but  not  a  significant 
predictor  for  conflicting  situation  time  (p  =  0.12)  and  baseline  situation  time  (p  =  0.33) 
(see  Figure  12). 
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Figure  12.  Regression  Plot  for  Scenario  Difficulty  for  Adjoined  Times. 

For  additional  time,  baseline  information  was  significantly  less  than 
conflicting  information  (p  <  0.01),  in  addition  to,  ambiguous/missing  information  being 
significantly  less  than  conflicting  information  {p  <  0.01).  However,  there  was  no 
significant  difference  between  ambiguous/missing  information  and  baseline  information 
( p  =  0.10).  Furthermore,  scenario  difficulty  was  a  significant  predictor  for 
ambiguous/missing  additional  time  (p  <  0.01)  and  conflicting  additional  time  (p  <  0.01), 
but  not  a  significant  predictor  for  baseline  additional  time  (p  =  0.07). 

For  time  to  decision,  ambiguous/missing  information  was  significantly 
greater  than  baseline  information  [p  <  0.01).  Conversely,  there  was  no  significant 
differences  between  ambiguous/missing  information  and  conflicting  information  (p  = 
0.10)  or  conflicting  information  and  baseline  information  (p  =  0.32)  (see  Figure  13). 
Interestingly,  scenario  difficulty  was  not  a  significant  predictor  for  time  to  decision. 
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ID# 

Y 

^ - Mean(  I  I  D,  Ambiguous/Missing) 

Q  - Mean(  I  I  D,  Baseline) 

^  - Mean(  I  I  D,  Conflicting) 

Figure  13.  Participant  Mean  Time  to  Decision  for  the  Categories  of  Uncertainty. 

c.  Interrelated  Times 

For  situation  time/additional  time,  the  researchers  found  significant 
differences  between  all  three  categories  of  uncertainty:  (1)  ambiguous/missing 
information  was  significantly  greater  than  conflicting  information  (p  <  0-01),  (2) 
ambiguous/missing  information  was  significantly  greater  than  baseline  information  [p  < 
0.01),  and  (3)  conflicting  information  was  significantly  greater  than  baseline  information 
{p  <  0.01).  For  time  to  decision/additional  time,  the  researchers  found  two  significant 
differences:  (1)  ambiguous/missing  information  was  significantly  greater  than  baseline 
information  (p  <  0.01)  and  (2)  conflicting  information  was  significantly  greater  than 
baseline  information  (p  <  0.01).  Additionally,  scenario  difficulty  was  a  significant 
predictor  for  both  ambiguous/missing  information  situation  time/additional  time  and 
ambiguous/missing  information  time  to  decision/additional  time.  However,  it  was  not  a 
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significant  predictor  for  conflicting  information  situation  time/additional  time,  conflicting 
information  time  to  decision/additional  time,  baseline  information  situation 
time/additional  time,  and  baseline  information  time  to  decision/additional  time. 

d.  Within  Category  Times 

For  ambiguous/missing  information,  the  researchers  found  a  significant, 
positive  correlation  between  situation  time  and  time  to  decision  (r  =  0.54,  p  <  0.01). 
However,  there  were  no  significant  correlations  between  situation  time  and  additional 
time  ( r  =  0.34,  p  =  0.08)  and  between  additional  time  and  time  to  decision  (r  =  0.22,  p  = 
0.24). 

For  conflicting  information,  the  researchers  found  two  significant 
correlations:  (1)  situation  time  and  time  to  decision  (r  =  0.64,/?  <  0.01)  and  (2)  additional 
time  and  time  to  decision  (r  =  0.50.  p  <  0.01).  However,  no  significant  correlations  were 
found  between  additional  time  and  situation  time  (r  =  0.26,  p  =  0.17). 

For  baseline  information,  the  researchers  found  significant,  positive 
correlations  between  all  three  time  segments:  (1)  situation  time  and  additional  time  (r  = 
0.49,  p  <  0.01),  (2)  situation  time  and  time  to  decision  ( r  =  0.53,  p  <  0.01),  and  (3) 
additional  time  and  time  to  decision  (r  =  0.5 1  ,p  <  0.01). 

B.  SUMMARY  STATISTICS  FOR  UNCERTAINTY  RESPONSE  SCALE 

URS  data  was  produced  through  a  scoring  system  on  a  Likert  Scale,  thus  non- 
parametric  tests  (Spearman’s  Rho)  were  used  to  determine  correlation.  Additionally, 
descriptive  statistics  provide  means,  medians,  and  standard  deviations  for  each  category 
of  the  URS.  For  this  study,  EU  served  as  the  primary  metric  of  analysis.  CU  and  DFC 
were  recorded  for  supplementary  analysis  and  future  research. 

1.  Descriptive  Statistics  for  URS 

CU  had  the  highest  mean  score  of  62  (SD  =  8),  while  EU  had  the  lowest  mean 
score  of  32  (SD  =  7).  Additionally,  DFC  produced  a  mean  score  of  59  (SD  =  7).  The 
median  score  for  CU  was  64,  while  the  median  score  for  EU  was  33.  DFC  had  a  median 
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score  of  58.  The  small  standard  deviations  coupled  with  the  medians  being  relatively 
close  to  the  means  suggest  a  small  amount  of  variance  amongst  the  URS  scores  (see 
Table  5). 


Category 

EU 

DFC 

CU 

Mean 

32 

59 

62 

Standard  Error 

1 

1 

2 

Median 

33 

58 

64 

Standard  De viation 

7 

7 

8 

Sample  Variance 

50 

55 

65 

Kudos  is 

0 

2 

0 

Skewness 

0 

-1 

0 

Range 

29 

37 

31 

Minimum 

19 

37 

45 

Maximum 

48 

74 

76 

Table  5.  Descriptive  Statistics  for  URS. 

2.  Inferential  Statistics  for  URS 

The  researchers  found  no  significant  correlations  between  the  categories  of  the 
URS:  (1)  CU  and  EU  (p  =  0.15,/?  =  0.44),  (2)  CU  and  DFC  (p  =  -0.22 ,p  =  0.25),  and  (3) 
EU  and  DFC  (p  =  -0.30 ,p  =  0.12).  These  findings  suggest  that  the  URS  measures  distinct 
characteristics  or  variables,  and  are  not  associated  or  related  to  one  another. 

C.  SUMMARY  STATISTICS  FOR  EXPERIENCE 

Operational  experience  was  discussed  in  Chapter  II. C. 3.  However,  a  more 
detailed  analysis  is  warranted  due  to  the  inclusion  of  experience  as  a  hypothesis. 
Consequently,  descriptive  statistics  will  be  provided  in  this  section. 

1.  Descriptive  Statistics  for  Operational  Experience 

The  participants  had  a  mean  of  12  ( SD  =  6)  months  of  operational  experience  in 
Iraq  and  a  mean  of  3  (SD  =  5)  months  of  experience  in  Afghanistan.  However,  the 
median  months  in  Afghanistan  are  0,  while  the  median  months  in  Iraq  are  12.  The  highest 
number  of  months  of  operational  experience  in  either  theatre  is  25  months.  The  variance 
for  experience  in  each  theatre  can  be  seen  in  Figure  14. 
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Nineteen  of  the  28  participants  did  not  serve  in  Afghanistan.  Three  participants 
had  not  served  in  Iraq,  and  6  participants  served  in  both  theatres.  Overall  operational 
experience  had  a  mean  of  14  months  ( SD  =  7),  and  a  median  of  12.5  months  (see  Figure 
15). 


10  15  20  25 

ID# 
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* - Time  in  Afghanistan  in  months 

0  — 'Time  in  Iraq  in  months 


Figure  14.  Participants  Months  of  Operational  Experience  by  Theatre. 


Figure  15.  Distribution  of  Total  Operational  Experience  in  Months. 
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D.  SUMMARY  STATISTICS  FOR  VARIABLE  ASSOCIATIONS 

The  researchers  found  no  significant  correlations  between  categories  of 
uncertainty  for  overall  total  time,  the  categories  of  the  URS,  and  operational  experience. 
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V.  DISCUSSION 


The  discussion  is  divided  into  three  parts:  (a)  implications  of  analysis,  (b) 
constraints  and  limitations,  and  (c)  conclusions  and  recommendations.  Part  A  will  the 
implications  of  the  statistical  findings  presented  in  IV.  Additionally,  it  will  outline  where 
and  possibly  why  those  statistical  findings  emerged.  Part  B  will  discuss  the  constraints 
and  limitations  of  the  research  based  on  the  data  from  Chapter  III  and  Chapter  IV.  It  will 
discuss  important  observations  obtained  from  the  entire  body  of  the  researchers’  work. 
Additionally,  it  will  describe  how  our  research  relates  to  the  prior  work  described  in 
Chapter  II  and  Chapter  III.  Finally,  Part  C  will  discuss  the  short-term  and  long-tenn 
vision  of  research  in  decision  making  under  uncertainty. 

A.  IMPLICATIONS  OF  ANALYSIS 
1.  Hypothesis  One 

Hai:  There  is  a  significant  difference  in  the  overall  total  time  with  respect 
to  the  three  categories  of  uncertainty. 

The  results  support  the  alternative  hypothesis  that  there  are  significant  differences 
between  the  categories  of  uncertainty  with  respect  to  overall  total  time.  The  results 
suggest  that  decision  makers  respond  slower  to  situations  with  ambiguous/missing 
information  compared  to  baseline  information  or  conflicting  infonnation.  Decision 
makers  respond  in  statistically  the  same  amount  of  time  when  faced  with  conflicting 
information  or  baseline  infonnation.  These  findings  produced  a  subsequent  set  of 
questions  concerning  the  nature  of  these  differences. 

The  differences  between  ambiguous/missing  information  and  baseline 
information  were  found  in  situation  time,  time  to  decision/additional  time,  time  to 
decision,  and  scenario  difficulty.  The  results  suggest  that  decision  makers  spent  more 
time  in  the  situation  time  and  time  to  decision  segments  for  ambiguous/missing 
information  compared  to  baseline  infonnation.  As  expected,  ambiguous/missing  situation 
time  and  ambiguous/missing  time  to  decision  were  positively  correlated.  Thus,  if  a 
decision  maker  spent  a  large  amount  of  time  on  situation  time,  then  he/she  would  spend  a 
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large  amount  of  time  on  time  to  decision,  and  visa  versa  (see  Figure  16).  Finally,  the 
results  suggest  that  scenario  difficulty  was  a  determining  factor  in  response  time  (i.e.,  the 
higher  the  level  of  difficulty,  the  slower  the  individual  responded  to  ambiguous/missing 
scenarios).  The  level  of  difficulty  was  not  a  significant  factor  for  baseline  overall  total 
time. 


Ambiguous-Missing  Information  >  Baseline  Information 


Supplementary  Note 


Ambiguous-Missing  TTDAT  >  Baseline  TTDAT 

Ambiguous-Missing  ST  is  positively  correlated  to  Ambiguous-Missing  TTD 


Figure  16.  Diagram  of  Statistical  Inferences  for  Differences  between 
Ambiguous/Missing  Information  and  Baseline  Information. 

The  differences  between  ambiguous/missing  information  and  conflicting 
information  for  decision  time  and  scenario  difficulty  are  additional  time  and  situation 
time.  The  results  suggest  that  decision  makers  spend  more  time  on  ambiguous/missing 
additional  time  compared  to  conflicting  additional  time.  Additionally,  the  results  suggest 
that  decision  makers  spend  more  time  on  ambiguous/missing  situation  time  compared  to 
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conflicting  situation  time.  Furthennore,  both  conflicting  situation  time  and  additional 
time  were  positively  correlated  to  time  to  decision.  Thus,  if  a  decision  maker  spends  a 
large  amount  of  time  on  either  conflicting  situation  time  or  additional  time,  he/she  would 
spend  a  large  amount  of  time  on  time  to  decision.  And  as  in  the  previous  paragraph, 
scenario  difficulty  was  a  predictor  for  ambiguous/missing  overall  total  time,  but  not  a 
predictor  for  conflicting  overall  total  time  (i.e.,  scenario  difficulty  had  no  effect  on 
conflicting  overall  total  time,  but  did  have  an  effect  on  ambiguous/missing  overall  total 
time). 

Finally,  the  results  suggest  that  if  a  decision  maker  is  slow  to  respond  on  any  of 
the  three  categories  of  uncertainty,  then  he/she  will  be  slow  on  the  others.  Thus,  the 
researchers  can  infer  that  the  response  time  to  any  of  the  categories  of  uncertainty  is 
correlated  with  a  decision  maker’s  response  time  to  others. 

a.  Ambiguous/Missing  Overall  Total  Time  and  Baseline  Overall 
Total  Time 

While  the  four  preceding  paragraphs  describe  “where”  the  differences  lie, 
an  explanation  of  “why”  is  needed.  To  address  the  differences  in  ambiguous/missing 
overall  total  time  and  baseline  overall  total  time,  the  researchers  offer  three  possibilities: 
(1)  the  differences  are  a  result  of  the  relationship  between  uncertainty  and  difficulty,  (2) 
the  ambiguous/missing  information  presented  in  the  additional  information  (AI)  did  not 
have  any  effect  because  the  decision  maker  had  thoroughly  evaluated  the  situation 
information  (SI)  and  produced  multiple  COAs,  and  (3)  the  ambiguous/missing 
information  in  the  AI  acted  as  a  retarding  agent  when  merged  with  the  situation 
information  (SI). 

As  noted,  the  researchers  attempted  to  evenly  distribute  the  infonnation  in 
both  the  additional  time  and  situation  time  sections.  Thus,  one  portion  of  the  slide  would 
not  possess  more  or  less  information  than  another  section  of  the  slide.  The  researchers  did 
conduct  reliability  analysis  to  determine  the  difficulty  of  each  scenario  (see  Chapter 
III.B.4).  They  did  not  conduct  reliability  analysis  on  the  complexity.  This  is  where  the 
discussion  about  the  relationship  between  complexity  and  difficulty  becomes  relevant. 
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Because  our  reliability  inquired  about  the  difficulty  of  the  situation,  and  the  not 
complexity,  these  results  can  only  address  the  former.  With  that  said,  difficulty  is  created 
by  numerous  factors,  including  context,  uncertainty,  task,  and  mission  type. 

It  is  not  clear  whether  uncertainty  produces  difficulty  or  difficulty 
produces  uncertainty.  As  previously  stated,  conflicting  information  emerges  when  two  or 
more  sources  or  information  contradict  one  another.  Conversely,  ambiguous/missing 
information  can  be  available  or  unavailable  to  the  decision  maker.  If  available  it  may  be 
unclear  to  its  meaning  or  relevance  to  the  mission;  if  unavailable  the  decision  maker  is 
unable  to  access  it. 

To  create  an  ambiguous/missing  situation,  the  researchers  had  to  draft  a 
scenario  with  detailed  information  or  an  abundance  of  resources,  which  produced 
numerous  alternatives  for  the  decision  maker  to  compare  and  contrast.  The  decision 
maker  may  spend  more  time  reasoning  about  the  situation,  comprehending  the  facts,  and 
gaining  SA  (hence,  the  difficulty  previously  noted).  In  terms  of  this  study,  SI  was  clear 
and  concise,  providing  a  context  in  which  actors  were  defined  and  environments 
understood.  The  AI  provided  the  uncertainty.  Thus,  the  main  contributor  to  the 
differences  in  situational  time,  time  to  decision,  and  overall  total  time  could  be  the  nature 
of  the  AI. 

A  second  possibility  is  that  if  a  decision  maker  spent  a  large  amount  of 
time  reasoning  about  the  SI  (understanding  his  overarching  mission  and  how  it  relates  to 
the  situation),  then  he/she  may  have  spent  a  large  amount  of  time  producing  a  COA, 
regardless  of  the  AI,  in  the  time  to  decision  segment.  This  may  account  for  why 
additional  time  was  not  statistically  significant  between  ambiguous/missing  information 
and  baseline  information  (i.e.,  the  decision  maker  had  already  begun  reasoning  about  the 
situation  infonnation).  Thus,  the  decision  maker  had  already  developed  COAs,  and  the 
ambiguous/missing  information  presented  reinforced  the  rationale  of  at  least  one  of  the 
COAs,  or  possibly,  the  uncertainty  had  little  or  no  impact  on  his  decision-making 
process. 
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This  argument  has  implications  for  Klein’s  (1993)  RPD  model,  where  a 
decision  maker  relies  on  past  experience  when  choosing  a  COA.  Because  our  sample 
contained  individuals  who  had  operational  experience  in  Iraq,  Afghanistan,  or  both,  they 
may  have  experienced  similar  situations  and  missions.  Thus,  they  executed  what  they 
knew. 

A  third  possibility  arises  when  SI  and  AI  are  combined.  Because  the 
ambiguous/missing  information  was  not  presented  until  the  decision  maker  requested  the 
AI,  and  SI  remained  present  on  the  slide,  the  AI  drastically  impacted  time  to  decision. 
This  is  supported  by  the  finding  that  ambiguous/missing  time  to  decision/additional  time 
is  significantly  greater  than  baseline  time  to  decision/additional  time  (p  <  0.01),  and  there 
was  no  significant  difference  between  ambiguous/missing  additional  time  and  baseline 
additional  time.  The  findings  suggest  that  the  ambiguous/missing  AI  acted  as  a  much 
more  effective  retarding  agent  when  merged  with  the  SI  compared  to  baseline 
information.  That  is,  even  if  the  situation  time  between  ambiguous/missing  and  baseline 
were  equal,  the  ambiguous/missing  information  in  the  AI  would  slow  the  decision 
maker’s  response  when  compared  to  baseline  information.  More  importantly,  scenario 
difficulty  was  a  predictor  variable  in  ambiguous/missing  situation  time.  This  suggests  that 
the  more  difficult  the  scenario,  the  longer  time  the  decision  maker’s  situation  time.  This 
is  not  true  for  conflicting  situation  time  or  baseline  situation  time.  Consequently,  the 
researchers  believe  the  decision  maker  had  to  re-evaluate  all  their  previous  assumptions 
and  beliefs,  and  produce  new  COAs  or  iterate  previous  plans  for  ambiguous/missing 
scenarios. 

The  third  argument  has  implications  for  the  DMSC.  As  previously 
discussed,  Shattuck  and  Miller  (2006)  stress  the  importance  of  promulgation  of 
information  and  feedback  loops  in  DMSC.  Consequently,  the  emergence  of 
ambiguous/missing  information  in  AI  may  alter  the  lens  of  the  decision  maker.  Thus, 
his/her  interpretation  of  ground  truth  is  no  longer  valid,  and  the  decision  maker  must 
recalibrate  his/her  lens  to  achieve  effective  SA.  In  many  ways,  the  third  argument  lends 
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itself  to  the  cliche,  “the  calm  before  the  storm,”  (i.e.,  the  commander  assumes  his  lens  is 
properly  calibrated  to  the  environment  only  to  be  rattled  by  the  emergence  of  uncertainty 
in  the  tactical  battle  space). 

b.  Ambiguous/Missing  Overall  Total  Time  and  Conflicting  Overall 
Total  Time 

While  the  differences  described  above  are  difficult  to  express,  the 
disparity  in  ambiguous/missing  overall  total  time  and  conflicting  overall  total  time  is 
even  more  muddled.  The  most  straightforward  way  to  discuss  the  differences  between 
ambiguous/missing  overall  total  time  and  conflicting  overall  total  time  was  to  examine 
why  conflicting  overall  total  time  was  significantly  less  than  ambiguous/missing  overall 
total  time. 

Once  again,  the  idea  of  difficulty  arises  when  discussing  the  differences  in 
additional  time.  It  appears  that,  situation  time  is  significantly  different  between  the 
categories  mostly  due  to  the  predictor  variable  of  difficulty.  While  this  reason  may  seem 
somewhat  simplistic,  it  follows  from  the  data  analysis  reported  in  the  previous  chapter 
(see  Figure  17). 
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Ambiguous-Missing  Information  >  Conflicting  Information 


_ Supplementary  Note 

Ambiguous-Missing  TTDAT  >  Conflicting  TTDAT 


Figure  17.  Diagram  of  Statistical  Inferences  for  Differences  between 
Ambiguous/Missing  Information  and  Conflicting  Information. 

Scenario  difficulty  was  a  predictor  for  both  conflicting  additional  time  and 
ambiguous/missing  additional  time.  At  a  qualitative  level,  the  researchers  noticed  the 
decision  makers  re-reading  the  lines  due  to  confusion  or  disbelief  (i.e.,  the  contradictory 
information  prompted  the  decision  maker  to  ensure  the  knowledge  previously  acquired 
was  accurate).  The  decision  maker  spent  a  great  deal  of  time  re-reading  and  reinterpreting 
his  previous  beliefs. 

Unfortunately,  this  does  not  adequately  explain  why  conflicting  situation 
time  and  additional  time  were  significantly,  positively  correlated  to  conflicting  time  to 
decision.  The  results  suggest  that  because  additional  time  and  situation  time  were  not 
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significantly  correlated,  there  was  no  association  between  the  two  variables  (i.e.,  just 
because  a  decision  maker  was  fast  in  situation  time,  does  not  imply  he/she  was  fast  in 
additional  time). 

The  researchers  support  the  findings  concerning  time  to 
decision/additional  time.  Unlike  ambiguous/missing  time  to  decision/additional  time  and 
baseline  time  to  decision/additional  time  where  the  results  were  not  significantly 
different,  the  findings  indicate  that  time  to  decision/additional  time  for 
ambiguous/missing  information  was  significantly  greater  than  time  to  decision/additional 
time  for  conflicting  information.  Contrary  to  the  third  possibility  discussed  in  Chapter 
V.A.I.,  the  researchers  suggest  that  the  true  difference  lies  in  the  SI  (i.e.,  the  uncertainty 
introduced  in  AI  retarded  both  situations  equally,  thus  the  time  differential  is  created 
from  the  aforementioned  difficulty  element).  The  significant  difference  for  additional 
time  between  the  categories  is  diminished  by  the  time  to  decision  between  the  categories 
(i.e.,  the  time  differential  that  favored  conflicting  additional  time  was  mitigated  by  the 
time  differential  favoring  ambiguous/missing  additional  time).  This  is  further  supported 
by  the  times  to  decision  for  ambiguous/missing  information  and  conflicting  information. 
Consequently,  the  researchers  suggest  that  situation  information  and  scenario  difficulty 
were  the  determining  factors  on  whether  or  not  the  decision  maker  responded  faster  to 
ambiguous/missing  or  conflicting  information. 

c.  Conflicting  Overall  Total  Time  and  Baseline  Overall  Total  Time 

In  this  section  the  researchers  explain  why  conflicting  overall  total  time 
and  baseline  overall  total  time  are  not  statistically  different.  Situation  time  between  the 
two  categories  is  statistically  different,  but  additional  time  was  not.  Moreover,  time  to 
decision  between  the  categories  was  not  significant,  but  conflicting  time  to 
decision/additional  time  was  significantly  greater  than  baseline  time  to 
decision/additional  time. 

While  examining  the  differences  between  the  categories  of  uncertainty  is 
necessary,  it  is  also  important  to  analyze  where  they  are  similar.  For  example,  there  was 
no  statistical  difference  between  conflicting  situation  time  and  baseline  situation  time. 
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Additionally,  scenario  difficulty  was  not  a  predictor  for  overall  total  time  for  either 
category.  These  two  findings  are  very  important.  The  researchers  can  suggest  that 
uncertainty  alone  does  not  detennine  the  rate  of  response,  but  rather  uncertainty  must  be 
placed  in  the  context  of  the  situation. 

The  differential  between  baseline  additional  time  and  conflicting 
additional  time  can  be  explained  in  a  manner  similar  to  ambiguous/missing  information 
and  conflicting  information.  When  the  decision  maker  is  presented  with  contradictory 
information,  the  researchers  often  observed  many  participants  re-reading  the  given 
information  to  reinforce  or  reconsider  their  previous  assumptions.  This  would  account  for 
the  statistical  significance  across  the  categories  of  uncertainty  found  in  additional  time. 

By  definition,  ambiguous/missing  information  presents  the  decision  maker 
with  more  alternatives  to  consider  compared  to  conflicting.  However,  an  argument  can 
also  be  made  that  baseline  information  is  not  much  different  than  conflicting  information 
(i.e.,  having  an  explicit  understanding  of  the  information  may  be  the  same  as  having  two 
explicit  understandings).  Consequently,  if  a  decision  maker  believes  both  to  be  equally 
valid  interpretations,  then  either  interpretation  is  adequate  and  could  be  assumed  to  be 
true.  This  may  explain  why  there  is  no  statistical  difference  for  time  to  decision  when 
comparing  baseline  and  conflicting  infonnation.  Moreover,  the  researchers  suggest  that 
the  time  to  decision/additional  time  difference  found  is  not  large  enough  to  account  for 
the  non-significant  situation  time  difference.  Thus,  time  differentials  in  additional  time 
were  mitigated  by  situation  time  and  time  to  decision.  Interestingly,  the  findings  in  this 
section  support  the  findings  of  the  third  pilot  study  (see  Chapter  III.  A. 3). 

2.  2nd  Hypothesis 

Ha2a:  There  is  a  significant,  positive  correlation  between  emotional 
uncertainty  and  overall  total  time  for  ambiguous/missing  information  and 
conflicting  information. 

Ha2t>:  There  is  a  significant,  negative  correlation  between  cognitive 
uncertainty  and  overall  total  time  for  ambiguous/missing  information  and 
conflicting  information. 
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The  results  refute  the  alternative  hypothesis  that  cognitive  uncertainty  is 
significantly,  negatively  correlated  with  ambiguous/missing  overall  total  time  or 
conflicting  overall  total  time.  Additionally,  the  results  refute  the  alternative  hypothesis 
that  emotional  uncertainty  is  significantly,  positively  correlated  with  ambiguous/missing 
overall  total  time  or  conflicting  overall  total  time. 

a.  Cognitive  Uncertainty  and  Overall  Total  Time 

The  findings  for  cognitive  uncertainty  are  interesting.  The  characteristics 
of  environments  in  which  naturalistic  decision  making  occurs  (Orasanu  and  Connolly, 
1993)  are  prevalent  in  the  military.  However,  this  does  not  imply  that  structure  does  not 
exist.  Instead,  the  researchers  suggest  that  at  the  beginning  of  most  military  operations 
there  is  structure,  but  uncertainty  can  still  exist  in  the  battle  space.  Consequently,  it  is  not 
uncommon  for  structure  and  uncertainty  to  coexist  for  the  decision  maker. 

This  relationship  leads  to  anomalies  when  examining  cognitive  uncertainty 
and  response  time.  Because  the  military  places  a  great  deal  of  emphasis  on  C2, 
commanders  will  strive  to  avoid  organizational  chaos.  They  will  more  than  likely  have  an 
organizational  structure  that  is  accustomed  to  or  designed  to  mitigate  uncertainty.  This 
may  explain  why  cognitive  uncertainty  had  the  largest  average  score  compared  to  the 
other  categories  of  the  URS.  It  may  well  be  that  the  participants  were  accustomed  to 
operating  within  a  structured  environment.  Thus,  the  presence  of  uncertainty  is  not 
unexpected  or  foregone  to  their  cognitive  performance.  Instead,  it  is  assumed  to  be  a  part 
of  everyday  operation. 

b.  Emotional  Uncertainty  and  Overall  Total  Time 

The  lack  of  statistical  significance  for  emotional  uncertainty  and  overall 
total  time  can  be  explained  in  a  manner  similar  to  the  previous  discussions.  However,  it  is 
important  to  note  some  characteristics  of  the  emotional  uncertainty  data  prior  to 
proceeding  to  the  aforementioned  case.  Emotional  uncertainty  had  the  smallest  range  (R 
=  29)  and  variance  (v  =  50)  compared  to  the  other  URS  categories.  This  was  expected 
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because  the  sample  pool  consisted  of  military  officers  who  often  operate  in  dynamic, 
uncertain  environments.  Regardless,  the  researchers  were  somewhat  surprised  by  the 
findings. 

Maladaptive  behavior  toward  uncertainty  does  not  necessarily  lead  to  a 
failure  to  respond  quickly.  Although  a  decision  maker  may  experience  anxiety  or  sadness 
when  faced  with  uncertain  conditions,  he/she  may  still  be  able  to  make  a  quick  decision 
due  to  other  factors  overriding  their  emotional  response.  Future  research  must  be 
conducted  to  better  understand  why  emotional  uncertainty  does  not  directly  affect  the 
response  time  of  the  decision  maker. 

3.  3rd  Hypothesis 

Has:  There  is  a  significant,  negative  correlation  between  operational 

experience  and  overall  total  time  for  ambiguous/missing  infonnation  and 

conflicting  information. 

The  results  refute  the  alternative  hypothesis  that  operational  experience  is 
negatively  correlated  with  ambiguous/missing  overall  total  time  and  conflicting  overall 
total  time.  This  research  does  not  support  the  research  conducted  by  St.  John  et  al. 
(2000).  However,  it  is  important  to  note  St.  John  et  al.  (2000)  used  senior  and  junior 
ranks  to  conduct  their  study,  and  not  years  of  service.  However,  their  overarching 
assumption  was  that  higher  ranks  equal  more  experience. 

Once  again,  the  researchers  were  surprised  by  this  finding.  The  absence  of  a 
relationship  suggests  that  operational  experience  does  not  increase  the  speed  of  battle 
under  uncertainty  (i.e.,  more  operationally  experienced  decision  makers  do  not  make 
faster  decisions  under  uncertain  conditions,  nor  as  the  data  suggest,  do  they  make  slower 
decisions). 

Other  than  one  participant,  all  the  participants  had  at  least  six  months  of 
operational  experience.  The  researchers  omitted  all  participants  without  any  operational 
experience.  Consequently,  the  participants  operated  in  Afghanistan,  Iraq,  or  both.  The 
data  did  not  provide  the  researchers  with  the  ability  to  discriminate  perfonnance  based  on 
level  of  experience. 
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B.  CONSTRAINTS  AND  LIMITATIONS  OF  THE  RESEARCH 

The  researchers  experienced  two  challenges  during  the  study.  First,  defining  the 
relationship  or  association  between  complexity  and  difficulty  is  necessary  to 
understanding  uncertainty.  Second,  branch  participant  selection  is  important  due  to  the 
tasks  assigned  by  the  scenario.  As  discussed  previously,  the  relationship  between 
complexity  and  difficulty  is  important  in  terms  of  their  influence  on  uncertainty.  The 
researchers’  reliability  survey  inquired  only  about  difficulty  and  not  complexity.  These 
terms  are  most  likely  related  but  not  necessarily  synonymous.  Scenario  difficultly  has 
contributed  greatly  to  our  work  by  influencing  decision  time  across  the  categories  of 
uncertainty. 

While  the  relationship  of  complexity  and  difficulty  is  important,  participant 
selection  based  on  combat  branch  is  equally  vital  to  achieving  better  results.  While 
participants  were  procured  from  all  three  branch  categories  of  the  U.S.  Army,  the 
scenarios  were  constructed  from  a  CA  perspective  (i.e.,  the  situations  resembled  the  types 
of  tasks  and  mission  executed  by  infantry,  armor,  and  artillery  officers).  This  is  not  to  say 
that  the  CS  officers  or  CSS  officers  cannot  accomplish  the  missions  evaluated  by  the 
study.  In  fact,  the  combination  of  smaller,  lighter  forces  along  with  the  tactics  of  post 
September  11,  2001  insurgents  have  placed  many  CS  and  CSS  officers  in  direct  contact 
with  the  enemy.  Our  suggestion  points  out  that  the  branches  associated  with  CS  and  CSS 
receive  different  training  and,  for  the  most  part,  do  not  perform  enemy  engagement 
missions  as  frequently  as  their  CA  counterparts.  Consequently,  the  exclusion  of  CS  and 
CSS  officers  would  bring  greater  validity  to  the  results. 

C.  CONCLUSIONS  AND  RECOMMENDATIONS 

The  conclusion  and  recommendations  section  contains  three  parts:  (1) 
conclusions,  (2)  future  research,  and  (3)  summary.  The  conclusion  will  discuss  the 
important  information  ascertained  from  the  study,  while  future  research  will  offer 
suggestions  for  future  work  in  the  field  of  uncertainty.  Finally,  the  summary  discusses  the 
researchers’  final  thoughts  on  the  study. 
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1. 


Conclusions 


First,  the  evidence  suggests  that  decision  makers  respond  differently  when 
confronted  with  various  categories  of  uncertainty.  Additionally,  scenario  difficulty  was  a 
significant  factor  in  ambiguous/missing  information  response  time,  but  not  a  significant 
factor  in  determining  conflicting  or  baseline  information  response  time.  These  findings 
suggest  that  the  categories  of  uncertainty  coupled  with  scenario  difficulty  determine 
response  time.  However,  whether  or  not  these  differences  are  mostly  due  to  the  categories 
of  uncertainty,  scenario  difficulty,  or  both  could  not  be  determined  by  this  study. 
Consequently,  future  research  must  focus  on  teasing  out  these  differences  to  determine 
what  role  each  one  of  these  factors  plays  in  response  time. 

Second,  the  absence  of  an  association  between  operational  experience  and 
response  time  is  dubious.  However,  this  study  did  not  look  at  the  quality  of  decisions,  but 
focused  on  response  time.  The  true  differences  may  be  found  in  the  quality  or  type  of 
decision  vice  the  response  time.  This  theory  is  being  explored  by  TRAC-FLVN  in  an 
adjacent  study.  Third,  because  there  was  no  association  between  the  analyzed  categories 
of  the  URS  and  overall  total  time,  the  researchers  believe  calibration  may  have  been  the 
cause  (i.e.,  because  military  officers  operate  in  uncertain  environments,  they  have  learned 
how  to  mitigate  its  effects  to  remain  efficient).  However,  future  research  is  needed  to 
determine  if  this  theory  is  correct. 

2.  Future  Research 

The  conclusions  provide  a  wealth  of  questions  for  future  research.  These 
questions  can  be  grouped  into  four  areas  of  study:  (a)  additional  research  on  decision¬ 
making  response  to  uncertainty,  (b)  application  to  modeling  and  simulation  for  human 
behavior  in  software  agents,  (c)  expansion  to  military  services,  and  (d)  application  to 
other  fields  of  occupation. 

a.  Additional  Research  on  Decision-Making  Response  to 
Uncertainty 

This  study  explores  the  categories  of  uncertainty  in  tenns  of  response 

time.  However,  response  time  is  just  one  of  many  performance  measures  that  can  be 
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examined  to  determine  how  individuals  respond  under  uncertainty  in  tactical  situations. 
TRAC-FLVN  will  be  publishing  an  adjacent  report  on  this  study  that  classifies  the 
decisions  made  by  the  participants  in  tenns  of  passive,  cautious,  and  decisive  action.  This 
analysis  will  help  determine  what  types  of  decisions  individuals  make  when  faced  with 
the  different  categories  of  uncertainty.  Additionally,  research  is  needed  to  explore  the 
relationship  between  complexity  and  difficulty  and  their  impact  on  uncertainty.  By  doing 
so,  researchers  will  be  able  to  develop  methods  that  can  tease  out  the  influence  of  these 
constructs  and  provide  more  conclusive  findings. 

b.  Application  to  Modeling  and  Simulation  for  Human  Behavior 

This  study  provides  the  foundation  for  creating  software  agents  that  better 
model  individual  behavior  under  conditions  of  uncertainty.  Modeling  human  behavior  is 
a  difficult  task,  but  it  is  vital  to  understanding  decisions  made  at  the  tactical  level.  While 
various  types  of  uncertainty  lead  to  differences  in  response  time,  research  must  focus  on 
other  factors  that  may  also  affect  decision  makers.  The  injection  of  harsh  weather  or 
unbearable  heat  may  drastically  affect  an  individual’s  ability  to  reason  under  uncertainty. 
Acute  of  chronic  fatigue  may  also  affect  a  decision  maker’s  ability  to  reason  about 
uncertainty.  Data  that  result  from  the  interactions  of  these  factors  may  produce  more 
robust  algorithms  to  better  model  human  behavior. 

c.  Expansion  to  the  Military  Services 

The  scenarios  used  in  this  study  were  crafted  for  U.S.  Army  personnel. 
However,  uncertainty  is  not  only  found  in  U.S.  Army  operations,  but  in  other  military 
services  as  well.  Scenario  generation  should  be  developed  in  other  tactical  environments, 
including  continuous  operations  at  sea  and  air  defense  operations  at  home  and  abroad. 
For  example,  U.S.  Navy  carrier  strike  groups’  operations  routinely  deal  with  uncertainty 
when  coordinating  organic  and  non-organic  assets.  Additionally,  combat  operations,  and 
in  particular,  amphibious  operations,  conducted  by  the  U.S.  Marine  Corps  parallel  the 
tactical  and  operational  environments  experienced  by  those  in  the  U.S.  Army.  The 
challenge  is  to  develop  scenarios  that  provide  tactical  realism  and  meet  the  specific 
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human  behavior  modeling  needs  of  the  services.  An  even  more  challenging  approach 
would  be  to  inject  categorical  uncertainty  into  staff  operations  to  determine  how  it  affects 
team,  staff  planning  operations,  and  both  at  the  single  service  and  joint  levels. 

d.  Application  to  Civilian  Occupations 

While  modeling  military  operations  and  decision  makers  was  the  central 
focus  of  this  research,  examining  the  impact  of  uncertainty  on  occupations  outside  the 
military  may  also  prove  beneficial.  Businesses  routinely  operate  with  uncertainty  at  all 
levels.  For  example,  commodity  speculators  attempt  to  detennine  the  future  price  of  oil 
based  on  complex,  dynamic  demand  and  supply  issues  in  the  near  and  long  term. 
Additionally,  transportation  and  logistics  companies  face  uncertain  conditions  in  weather, 
prices,  and  interruptions  in  supply  lines  when  making  critical  operational  decisions  that 
have  long  term  consequences.  Developing  software  agents  that  mimic  human  behavior 
may  help  us  understand  these  uncertainties  by  properly  modeling  how  they  influence 
human  decision  making.  Such  research  could  provide  additional  insights  into  modeling 
and  simulation  techniques  and  how  decision  makers  outside  the  military  cope  with 
uncertain  conditions. 

3.  Summary 

Research  in  decision  making  under  uncertainty  has  great  potential.  The  study  of 
human  reasoning  under  uncertain  conditions  is  critical  to  understanding  the  dynamic  and 
unpredictable  nature  of  war.  The  researchers  are  confident  that  the  findings  in  this  study 
will  add  to  the  body  of  knowledge  in  this  field  of  decision  making  under  uncertainty  and 
lead  to  improved  comprehension  of  the  battle  space.  Additionally,  these  proposed 
research  areas  will  provide  a  more  robust  understanding  of  uncertainty  in  modeling  and 
simulation.  The  results  of  the  research  coupled  with  the  improved  modeling  and 
simulation  will  allow  commanders  to  eventually  view  the  battle  before  it  is  fought.  These 
improvements  have  the  potential  to  revolutionize  the  manner  in  which  military  leaders 
train  for,  reason  about,  and  conduct  military  operations  at  all  levels  of  war. 
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APPENDIX  A.  UNCERTAINTY  RESPONSE  SCALE  TEST 


I  •  1  ‘end  to  give  up  easily  when  I  don’t  clearly  understand  a  situation. 


-L 


_L 


A 


Never  Sometimes  Nowand  Often  Always 
Then 

2.  When  1  go  shopping,  I  like  to  have  a  list  exactly  of  what  I  need. 


_L 


A 


Never  Sometimes  Now  and  Often  Always 
Then 


3.  I  feel  better  about  myself  when  I  know  that  I  have  done  all  I  can  to  accurately  plan  ray  future 


1 


Never  Sometimes  Nowand  Often  Always 
Then 


4.  Sudden  changes  make  me  feel  upset. 


1 


A 


Never  Sometimes  Nowand  Often  Always 
Then 

^  W  hen  making  a  decision,  I  am  deterred  by  the  fear  of  making  a  mistake. 


1 

_2 _ 

3 

4 

5 

Never 

Sometimes 

Now  and 
Then 

Often 

Always 

hen  uncertain,  I  act  very  cautiously  until  I  have  more  information  a 

1 

2 

.  3 

4 

5 

Never 

Sometimes 

Now  and 
Then 

Often 

Always 
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7.  I  like  to  have  things  under  control. 


1 

2 

3 

4 

5 

Never 

Sometimes 

Now  and 
Then 

Often 

Always 

8.  When  the  future  is  uncertain,  1  generally  expect  the  worst  to  happen. 

1 

2 

3 

4 

Never 

Sometimes 

Now  and 
Then 

Often 

Always 

9.  Facing  uncertainty  is  a  nerve 

■wracking  experience. 

1 

2 

3 

4 

5 

Never 

Sometimes 

Now  and 
Then 

Often 

Always 

10. 1  get  worried  when  a  situation  is  uncertain. 

1 

2 

3 

4 

_ 1 

Never 

Sometimes 

Now  and 
Then 

Often 

Always 

11.  Thinking  about  uncertainty  makes  me  feel  depressed. 

1 

2 

3 

4 

_ i 

Never 

Sometimes 

Now  and 
Then 

Often 

Always 

12. 1  find  the  prospect  of  change  exciting  and  stimulating. 

1 

2 

3 

4 

_ 5 

Never 

Sometimes 

Now  and 
Then 

Often 

Always 

13.  Uncertainty  frightens  me. 

1 

2 

3 

4 

5 

Never 

Sometimes 

Now  and 

Often 

Always 

Then 
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14.  There  is  something  exciting  about  being  kept  in  suspense. 


- — *■ - ±-  J _ 

Never  Sometimes  Now  and 
Then 

Often 

_ 2 

Always 

15.  The  idea  of  taking  a  trip  to  a  new  country  fascinates  me. 

_ I  2 _ 2_ 

4 

5 

Never  Sometimes  Now  and 
Then 

Often 

Always 

16. 1  like  going  on  holidays  with  nothing  planned  in  advance 

_ 1 _ 2  3 

4 

5 

Never  Sometimes  Now  and 
Then 

Often 

Always 

17. 1  think  you  have  to  be  flexible  to  work  effectively. 

- 1 _ 2_  3 

4 

5 

Never  Sometimes  Now  and 
Then 

Often 

Always 

1 8.  Taking  chances  is  part  of  life. 

- 1 _ 2 _ 3 

4 

Never  Sometimes  Now  and 
Then 

Often 

Always 

19.  When  I  feel  uncertain  about  something.  1  try  to  rationally  weigh  up  all  the  information  I  have. 

- 1 - 2 _ 2 _ 4 _ 5 

Never  Sometimes  Now  and  Often  Always 

Then 

20.  Before  making  any  changes,  I  need  to  think  things  over  thoroughly. 

- 1 _ 2 _ 3 _ 4 _ $ 

Never  Sometimes  Now  and  Often  Always 

Then 
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21.|  prefer  to  stick  to  tried  and  tested  ways  of  doing  things. 


1 

_ 2 _ 2 _ * - 

_ 5 

Never 

Sometimes  Now  and  Often 

Then 

Always 

22. 1  like  to  have  my  weekends  planned  in  advance. 

1 

_2 _ 3 _ 4 - 

5 

Never 

Sometimes  Now  and  Often 

Then 

Always 

23. 1  feel  curious  about  new  experiences. 

1 

_2 _ 2 _ 4 - 

_ 5 

Never 

Sometimes  Now  and  Often 

Then 

Always 

24. 1  like  to  think  of  a  new  experience  in  terms  of  a  challenge. 

1 

2 _ 2 _ 4 - 

_ i 

Never 

Sometimes  Now  and  Often 

Then 

Always 

25.  A  new  experience  is  an  occasion  to  learn  something  new. 

1 

_ 2 _ 2 _ 4 _ 

_ J 

Never 

Sometimes  Now  and  Often 

Then 

Always 

26.  When  I  feel  a  situation  is  unclear,  I  try  to  do  my  best  to  resolve  it 

1 

_J _ 1 _ 4 - 

5 

Never 

Sometimes  Now  and  Often 

Then 

Always 

27. 1  like  to  know  exactly  what  I’m  going  to  do  next. 

1 

2  3  4 

5 

Never 

Sometimes  Now  and  Often 

Then 

Always 
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28  the  best*01"8  UnCCrtai"  situation> 1  ,cnd  t0  P^parc  as  much  as  possible,  and  then  hope  for 


J. 


_2_ 


_1 


Never  Sometimes  Nowand  Often  Always 
Then 

29. 1  feci  relieved  when  an  ambiguous  situation  suddenly  becomes  clear. 


_L 


JL 


A. 


A 


Never  Sometimes  Now  and  Often  Always 
Then 

30.  When  I  feel  uncertain,  I  try  to  take  decisive  steps  to  clarify  the  situation. 


1 


2_ 


Never  Sometimes  Nowand  Often  Always 
Then 


31.  When  I  can’t  dearly  discern  situations,  I  get  apprehensive. 


1 


± 


A 


Never  Sometimes  Nowand  Often  Alwavs 
Then 


32. 1  enjoy  finding  new  ways  of  working  out  problems. 


L 


2. 


A. 


A 


Never  Sometimes  Now  and  Often  Always 
Then 

33.  When  I’m  not  certain  about  someone's  intentions  towards  me,  I  often  become  upset  or  angry 


I 


A 


Never  Sometimes  Now  and  Often  Always 

Then 

34.  New  experiences  can  be  useful. 

- 1 _ 2 _ 2 _ 4 _ 5 

Never  Sometimes  Now  and  Often  Always 

Then 
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35.  When  uncertain  about  w  hat  to  do  next  I  tend  to  feel  lost. 


Never  Sometimes 

Now  and 
Then 

Often 

Always 

36. 1  feel  anxious  when  things  are  changing. 

1  2 

3 

4 

5 

Never  Sometimes 

Now  and 
Then 

Often 

Always 

37.  New  experiences  excite  me. 

1  2 

3 

4 

Never  Sometimes 

Now  and 
Then 

Often 

Always 

38. 1  think  variety  is  the  spice  of  life. 

1  2 

3 

4 

5 

Never  Sometimes 

Now  and 
Then 

Often 

Always 

39. 1  try  to  have  my  life  and  career  clearly  mapped  out. 

1  7 

3 

4 

5 

Never  Sometimes 

Now  and 

Often 

Always 

Then 


40. 1  think  a  mid-life  career  change  is  an  exciting  idea. 

1 _ l _ l _ i — 

Never  Sometimes  Now  and  Often 

Then 

41.  When  a  situation  is  unclear,  it  makes  me  feci  angry. 

1  2 _ 3 _ 4 - 1 

Never  Sometimes  Now  and  Often  Always 

Then 


__S 

Always 
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42. 1  enjoy  unexpected  events. 


_L 


3 


Never  Sometimes  Now  and 
Then 


Often 


Always 


43. 1  like  things  to  be  ordered  and  in  place,  both  at  work  and  at  home. 


2_ 


1 


Never  Sometimes  Now  and  Often  Always 
Then 

44. 1  really  get  anxious  if  I  don't  know  what  someone  thinks  about  me. 
- ! - 2 - J _ 4 _ 5 


Never  Sometimes  Now  and 
Then 


Often 


45. 1  easily  adapt  to  novelty. 

_ ! _ 2_ 


Never  Sometimes  Now  and  Often 
Then 

46. 1  am  hesitant  when  it  comes  to  making  changes 


Always 


1 


Always 


1 


JL 


Never  Sometimes  Now  and 
Then 


Often  Always 

47. 1  like  to  plan  ahead  in  detail  rather  than  leaving  things  to  chance. 


_ ! _ 2 _ 2 _ 4 _ 5 

Never  Sometimes  Now  and  Often  Always 

Then 

48.  Before  I  buy  something,  I  have  to  view  every  sample  I  can  find. 

- 1 _ 2 _ l _ 4 _ 5 

Never  Sometimes  Now  and  Often  Always 

Then 
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APPENDIX  B.  MODIFIED  CDM  INTERVIEW  SHEET 


Participant  Name:  |Event# 

Recall  a  tactical  situation  during  a  recent  deployment  to  a  combat  zone  In  which  you  were  confronted  with 
uncertainty  and  had  to  make  a  decision.  Uncertainty  can  be  described  simply  as  a  sense  of  doubt  that  blocks  or 
delays  action.  There  are  no  right  or  wrong  answers  and  we  are  in  no  way  evaluating  your  performance  or  the 
decisions  you  made. 

Critical  incident  summary 
(Location,  Date/Time,  Key 
Personnel,  General  Description) 

Detailed  Incident  timeline 
(Sequence  of  key  incidents, 
activities) 

Identify  uncertainty  (uncertainties) 
in  the  timeline 

Describ e  th  e  specific  natu  re  of  th e 
uncertainty  (what  was  known  by 
whom,  what  was  not  known  by 
whom,  what  the  participant 
wanted/needed  to  know  in  that 
context) 

Describe  the  actions  that  were 
undertaken  to  overcome  the 
uncertainty  (e.g.,  asked  for 
additional  data  or  Intel,  relied  on 
previous  experiences,  etc.) 

Describe  the  effect  of  the 
uncertainty  on  the  decision  that 
was  made 

Describe  what  you  would  do 
differently  if  faced  with  the  same 
situ  ation 

Table  6.  CDM  Uncertainty  Elicitation  Interview  Sheet. 
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APPENDIX  C.  NPS  PILOT  STUDY  #  1  SCREEN  SHOTS 


TARGET  SET 


dog 

bog 

hog 

nog 

log 

rog 

PROCEED 


Figure  18.  Low  Resolution  Ambiguity  Slide. 


TARGET  SET 


tac 

tan 

tab 

tag 

tam 

tap 

PROCEED 


Figure  19.  Medium  Resolution  Ambiguity  Slide. 
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TARGET  SET 


rake 

rate 

rare 

raze 

race 

rave 

PROCEED 

WORD  17  of  30 

► 

Figure  20.  High  Resolution  Ambiguity  Slide. 
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APPENDIX  D.  NPS  PILOT  STUDY  #  2  SCREEN  SHOTS 


Figure  2 1 .  Conflicting  Information  Slide. 


Figure  22.  Missing  Information  Slide. 
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Figure  23.  Ambiguous  Information  Slide. 


Figure  24.  Baseline  Information  Slide. 
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APPENDIX  E.  CGSC  TACTICAL  VIGNETTES 


Situation 

a.  You  are  a  MITT  team  leader  advising  an 
Iraqi  BN. 

b.  Insurgents  have  been  operating  in  the 
area  to  include  emplacement  of  lEDs 
and  ambush  sites. 

c.  The  MSR  has  been  swept  for  lEDs 
<®MfesConMtePo«)  w  ithi  n  the  last  (2)  hours  and  there  is  no 

_4 _ I _ | _ | _ |  evidence  of  lEDs  or  ambush. 

2  3  4  5  ™  Additional  Information 

1 .  Near  dark,  the  Iraqi  BN  Comm  ander  receives  a  report  of  (5)  dead  bodies 
along  the  MSR  at  Point  A. 

2.  You  accom  pany  the  Iraqi  CDR  and  a  heavily  armed  Iraqi  Police  (IP)  escort 
south  along  the  ASR  to  Point  A. 

3.  While  enroute,  in  the  vicinity  of  CKP  1,  you  receive  a  radio  report  directly 
from  your  higher  HQ  that  the  dead  bodies  are  located  on  the  MSR  at  Point 
B,  to  your  rear,  in  a  location  you  have  already  driven  past. 


Suspected  Dead 
Bodies  (2d  Report) 


PLEASE  SNAP  YOU  R  CLIC  KER  WH  EN  YOU  HAVE  MADE  ADECISION 

SLIDE  1 


Figure  25.  Conflicting  Information  Slide. 


Situation 

a.  You  a  re  a  S  pecia  I  Forces  ODA  team  lead  er 
operating  with  (5  0) freedom  fighters.  Your 
split  ODA  team  s  (mounted)  are  occupying  the 
(2)  ABF  positio  ns. 

b.  Your  mission  is  a  coordinated  daylight  attack 
against  the  rem  nants  of  a  light  infantry  BN 
(minus) with  approximately  (200)  entrenched 
ona  hillside. 

c.  The  enemy  BN  has  mortarsand  small  arms 
capability. 

d.  Prior  to  the  attack,  an  a  ir  strike  will  reduce 
the  enemy  BN  strength  to  an  acceptable 
attac  k/defend  ratio. 


Additional  Information 

1.  You  observe  several  rounds  “off  target”  during  the  air  strike. 

2.  The  adjacent  units  report  they  started  their  attacks. 

3.  You  do  not  have  any  inform  ation  about  the  remaining  enemy  strength. 


PLEASE  SNAP  YOU  R  CLICKER  WHEN  YOU  HAVE  MADE  A  DECISION 

SLIDE  2 


Figure  26.  Ambiguous/Missing  Information  Slide. 
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Situation 


You  are  leading  an  infantry  platoon  raid  to 
detain  a  suspected  anti-coalition  leader 
running  a  weapon’s  black  market  in  OBJ 
Able . 

A  platoon  blocking  force  is  moving  north  on 
Route  Black,  and  will  link  ip  with  your 
assa  ult  force  to  secure  the  o  bje ctive. 

The  two  platoons  have  enough  combat 
power  to  secure  al I  ro  ute  s  lead  i ng  o  ut  of  the 
objective. 


Additional  Information 

1 .  The  blocking  force  reports  being  set  in  position  to  the  South  and  all  escape 
routes  are  secured. 

2.  You  continue  to  OBJ  Able,  enter  crowded  market  conditions,  and  observe 
the  anti-coalition  leader  in  a  dark  blue  BMW  sedan. 

3.  The  dark  blue  BMW  sedan  is  unable  to  move  due  to  congestion  in  the 
market  area. 

PLEASE  SNAP  YOUR  CLICKER  WHEN  YOU  HAVE  MADE  A  DECISION 

SLIDE  3 


Figure  27.  Baseline  Information  Slide. 


S  ituation 

a.  You  a  re  a  C  O  CD  R  in  charge  of  a  re  mote 
FOB. 

b.  You  have  a  three  vehicle  HMMWV  QRF 
availab  le. 

c.  There  a  re  (2)  NAIs  within  a  5  KM  range  that 
show  evidence  ofrecentoccupation  and 
discarded  military  equipment:  (1)  a  hilltop  to 
the  North  (NAI  1)  with  some  hardened 
buildings,  and  (2)an  abandoned  airport  to  the 
South  (NAI  2  ).  Bo  th  site  s  have  been  used  to 
launch  mortar  attacks  a  gainst  the  FOB. 

d.  The  CO  usually  responds  to  indirect  fire  by 
locating  and  destroying  the  mortar  sites. 

Inform  ation 

1 .  At  0  1  05  Hrs  ,  you  are  located  in  your  CO  TOC,  and  hear  the  sound  of 
incoming  in  d  ire  ct  fire .  Th  e  first  round  lands  within  50  meters  of  the  TOC. 

2.  Your  1  SG  tells  you  that  a  PSG  and  himself  believe  the  round  cam  e  from 
the  South. 

3.  Your  XO  and  a  guard  on  the  roof  report  they  believe  the  round  came  from 
the  North . 

PLEASE  SNAP  YOUR  CLICKER  WHEN  YOU  HAVE  MADE  A  DECISION 

SLIDE  4 


Figure  28.  Conflicting  Information  Slide. 
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Situation 

a.  You  are  the  NightOIC  responsible  for  the  Division 
ASP  security. 

b.  ASP  stocks  are  stored  on  the  ground  on  unimproved 
surfaces  and  cover  a  2km  x  2km  area. 

c.  (3)  Avengers  (ADA  Gun)  with  FUR  capability  are 

manned  to  support  security  operations . 

d.  A  dismounted  Q  RF  with  6  personnel  and  equipped 
with  small  arms  and  night  vision  devices  is  co¬ 
located  at  the  CP . 

e.  SOP  states  the  QRF  should  not  be  deployed  until 
enemy  activity  is  confirmed. 


Additional  Information 

1.  It  is  approximately  (5)  minutes  after  shift  change.  The  security  personnel 
were  just  replaced  and  should  be  heading  back  to  their  respective  platoon 
CPs. 

2.  A  Specialist  from  the  QRF  reports  movement  near  the  small  arms 
munitions  stocks.  The  Avenger  Crews  report  they  cannot  observe 
aforementioned  location. 

3.  Previously,  locals  have  looted  the  stocks  and  sold  ammunition  on  the 
black  market. 

PLEASE  SNAP  YOUR  CLICKER  WHEN  YOU  HAVE  MADE  A  DECISION 

SLIDE  5 


0  1  2  KM 


Figure  29.  Ambiguous/Missing  Information  Slide. 


Situation 

a.  You  are  the  OIC  responsible  for  convoy 
security. 

b.  During  convoy  operations,  a  local 
national  is  killed  in  a  fatal  car  accident 
involving  one  of  the  m  ilitary  convoy 
vehicles  near  CKP  2. 

c.  Battalion  SOP  and  TTP  is  to  remove 
military  and  civilian  bodies  to  a  Mortuary 
Affairs  team  co-bcated  at  an  Iraqi  Police 
Station  regardless  of  political  or  family 
affiliation. 

Additional  Information 

1.  You  secure  the  area  and  reportthe  accidentand  fatality  to  higher  HQ. 

You  identify  the  body  as  the  cousin  of  a  local  leader. 

2.  You  have  sufficient  combat  power  on  hand  to  both  secure  the  convoy  and 
escort  the  body  to  the  Mortuary  Affairs  team. 

3.  The  Mortuary  Affai rs  team  is  located  approximately  20  KM  south  of  the 
accident  site. 

PLEASE  SNAP  YOUR  CLICKER  WHEN  YOU  HAVE  MADE  A  DECISION 

SLIDE  6 


Figure  30.  Baseline  Information  Slide. 
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Situation 

a.  You  are  the  QRF  platoon  leader 
responding  to  a  report  of  dead  bodies 
along  the  MSR. 

b.  Insurgents  have  been  operating  in  the 
area  to  include  emplacement  of  lEDs 
and  am  bush  sites. 

c.  The  local  townspeople  are  sympathetic 
to  the  insurgents. 

d.  It  is  not  safe  to  be  in  this  area  after 
dark. 

Additional  Information 

1 .  You  find  (5)  dead  bodies  just  before  dark.  They  were  killed  execution  style. 

2.  A  local  reports  that  several  men  in  a  truck  dumped  the  bodies  approximately 
(1)  hour  earlier  and  then  continued  South. 

3.  A  different  local  indicates  the  bodies  were  executed  at  the  site  by  insurgents 
less  than  (30)  m  inutes  ago.  The  local  gives  you  an  exact  location  where  the 
insurgents  went  inside  the  town. 

PLEASE  SNAP  YOUR  CLICKER  WHEN  YOU  HAVE  MADE  A  DECISION 

SLIDE  7 


012345  KM 


Figure  3 1 .  Conflicting  Information  Slide. 


S  ituation 

a.  You  are  a  CO  C D R  c ond ucting  nightly  ambush 
missions  outside  the  FOB  to  destroy  enem  y  rocket 
launcher  team  s. 

b.  Your  CP  is  located  inside  the  nearby  town.  You 
have  (3)  platoons  available;  3RD  Platoon  is  the  main 
effort;  the  Battalion  Scout  platoon  is  occupying  the 
SBF  position. 

c.  CAS  (F-16's)  is  available  and  a  TACP  is  attached; 
Ap  ac  he  He  li co  p  te  rs  a  re  i  n  d  ire  ct  su  p  po  rt  o  f  the 
un  it’s  ope  ratio  n  s. 

d.  You  are  at  the  lim  it  of  yourcom  munications  ranges 
to  the  pi  ato  on  s  a  nd  to  h  ig  he  r  B  N  he  ad  qu  arte  rs  . 

Additional  Information 

1  .  You  receive  a  relayed  report  that  the  3rd  platoon  leader  has  been  trying  to 
reach  you.  3rd  platoon  is  reporting  contact  with  dismounts  N  orthwest  of 
their  location . 

2.  All  units  have  lost  radio  com  munications  with  the  Scout  Platoon  manning 
the  SBF  positbn.  Several  attem  pts  to  raise  them  on  the  radio  have  failed. 

3.  From  the  relayed  information  and  the  locations  on  the  FBCB2,  you  are  not 
sure  if  the  platoon  leader  is  actually  under  fire  from  insurgents  or  under 
attack  by  the  friendly  SBF  position. 

PLEASE  SNAP  YOU  R  C  LIC  KER  WH  EN  YOUHAVEMADE  A  DECISION 

SLIDE  8 


Figure  32.  Ambiguous/Missing  Information  Slide. 
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Situation 


Sicpciaus 

a. 

You  are  the  Night  Battle  Captain  in  the 

Activity 

BOC  and  are  responsible  for  F  OB 

\6 
\  Guard 
Tower  #2 

Tower#1 

security. 

* 

b. 

Your  BN  is  the  only  unit  operating  from 
this  FOB. 

\ 

fobT 

c. 

The  BN  QRF  consists  of  (4)  Bradley 

1 

d. 

Fighting  Vehicles  with  (4)  dismounts 
each. 

As  part  of  your  duties,  you  have  been 
given  authority  to  exercise  the  QRF  in 
support  of  FOB  security. 

Additional  Information 

1.  Guard  Tower  #1  reports  suspicious  enemy  activity  (lights/movem  ent)  1  KM 
Northwest  of  the  FOB  in  the  vicinity  of  the  road. 

2.  Guard  Tower  #2  confirms  observing  (2)  civilian  pickup  trucks,  and  reports 
observing  enemy  small  arms  tracer  fire  incoming  from  that  area. 

3.  A  guidons  call  and  a  status  check  confirms  that  no  units  are  currently 
operating  outside  of  the  FOB. 

PLEASE  SNAP  YOUR  CLICKER  WHEN  YOU  HAVE  MADE  A  DECISION 

SLIDE  9 


Figure  33.  Baseline  Information  Slide. 
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APPENDIX  F.  CGSC  STUDY  INSTRUCTIONS 


Operations  Research  Department  Lawrence  Shat  tuck,  Ph.D. 

Human  Systems  Integration  lgshattu@nps.edu 

Glasgow  Hall 
Naval  Postgraduate  School 
Monterey,  California  93943 

From:  LTJG  K.E.  Kemmerer,  USN 

To:  CGSC  Decision-Making  Participants 

Sbj :  DECISION-MAKING  STUDY  INSTRUCTIONS 

***P  LEASE  READ  THE  FOLLOWING  INSTRUCTIONS  ALOUD*** 

1.  You  are  about  to  participate  in  a  decision  making  study.  Please  read  these  directions 
carelully  and  aloud.  We  thank  you  in  advance  for  your  time  and  involvement. 

2.  This  study  consists  of  several  different,  independent  scenarios.  Each  scenario 
consists  oftwo  blocks  of  information:  (1)  situational  data  and  (2)  additional  data. 

You  will  read  both  the  situational  and  additional  data  aloud. 

3.  The  situational  information  provides  data  concerning  the  tactical  environment,  force 
positioning,  and  the  location  of  the  scenario.  After  you  read  the  situational  data 
and  are  ready  to  read  the  additional  data,  snap  the  clicker  providedto  you. 

The  additional  information  will  provide  real-time  data  describing  the  events  and 
intelligence  occurring  on  the  battlefield. 

4.  After  you  have  read  both  blocks  of  data  out  aloud,  you  will  be  asked  to  make  a 
decision.  Please  indicate  you  have  made  a  decision  by  snapping  your  clicker. 

5.  Once  you  snap  your  clicker,  you  will  be  asked  to  answer  two  questions: 

1)  Toprovidea  brief  description  of  your  decision? 

2)  How  did  you  arrive  at  your  decision? 

Please  do  not  snap  your  clicker  until  you  are  ready  to  answer  these  two 
questions  and  speak  clearly  into  the  microphone  while  doing  so. 

6.  After  you  have  answered  both  questions,  snap  your  clicker  to  proceed  to  the  next 
scenario. 

7.  Please  test  that  your  clicker  is  working  properly  now. 

Once  again,  thank  you  for  your  time,  and  please  answer  to  the  best  of  your  ability. 
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